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Abstract 


This  paper  examines  the  role  of  elaborations  in  learning  a  skill  from  an 
instructional  text.  Two  sources  of  elaborations  are  compared:  those  provided  by  the 
author  and  those  generated  by  the  learners  while  reading.  In  two  studies,  we  tested  the 
effectiveness  of  author-eiaborated  and  unelaborated  versions  of  a  manual  in  terms  of  how 
well  they  helped  subjects  acquire  and  perform  a  cognitive  skill  (learning  to  use  a 
personal  computer).  In  addition  to  manipulating  the  availability  of  author-generated 
elaborations,  we  also  manipulated  how  effectively  subjects  could  generate  elaborations. 
Some  subjects  were  given  advance  information  about  the  tasks  they  would  have  to 
perform  so  that  they  would  generate  more  specific,  task-related  elaborations  while 
reading.  Elaborations  facilitated  performance,  regardless  of  whether  they  were  author-  or 
reader-generated.  One  source  was  sufficient,  however:  author-provided  elaborations 
helped  only  when  subjects  had  no  advance  information  about  the  tasks.  Indeed, 
subjects  with  both  sources  of  elaborations  performed  worse  on  skill-performance  tasks 
than  subjects  with  only  one.  We  explain  this  performance  deficit  in  terms  of  ineffective 
strategies  for  coping  with  an  increased  cognitive  load  within  a  limited  study  time. 


An  important  question  to  both  memory  theorists  and  pedagogists  is  what  variables 
will  improve  the  learning  and  retention  of  written  information.  One  such  variable  that 

has  been  the  topic  of  considerable  speculation  and  research  is  the  effect  of  elaborations 
(Anderson  &  Reder  1979:  Reder  1976:  Reder  1979:  Weinstein  1978:  Mandl  &  Ballstaedt 
1981:  Mandl,  Schnotz  &  Tergan  1984:  Bransford  1979:  Chiesi.  Spilich  &  Voss  1979: 
Craik  &  Tulving  1975;  Reder  (in  press)).  In  the  view  of  most  researchers,  there  are 
several  reasons  why  elaborations  should  help  subjects  learn  and  remember  the  mam 
ideas  of  a  text.  Elaborations  provide  multiple  retrieval  routes  to  the  essential  information 
by  creating  more  connections  to  the  learner's  prior  knowledge.  If  one  set  of  connections 
is  forgotten,  it  may  be  possible  to  retrieve  the  desired  information  another  way.  Further, 
if  the  learner  forgets  an  important  point,  it  may  be  possible  to  reconstruct  it  from  the 
information  that  is  still  available. 

For  the  purposes  of  this  paper,  we  define  elaborations  as  pieces  of  information 
that  support,  clarify,  or  further  specify  the  main  points  (or  gist)  of  a  text.  Elaborations 
take  many  forms  including  examples,  details,  analogies,  warrants,  restatements  and 

deductions.  Elaborations  can  arise  from  two  distinct  sources:  first,  the  text  itself  can 
contain  elaborations  of  the  main  ideas  and  second,  the  reader  can  generate  them 
independently  while  reading.  Each  source  has  its  own  merits  and  drawbacks. 

Elaborations  provided  by  the  author  of  the  text  may  be  more  accurate  than  those  the 

reader  can  come  up  with,  since  the  author  is  presumably  more  knowledgeable  about  the 
topic.  On  the  other  hand,  the  reader's  own  elaborations  are  likely  to  be  more  relevant 
to  his  or  her  immediate  purpose  for  reading,  and  also  more  memorable  (Bobrow  & 
Bower  1969:  Rohwer  &  Ammon  1971;  Rohwer.  Lynch.  Levin  &  Suzuki  1967,  Rohwer. 
Lynch,  Suzuki  &  Levin  1967). 

There  has  been  ample  research  supporting  the  idea  that  reader-generated 


elaborations  facilitate  retention.  This  support  comes  from  experiments  where  subjects 
have  additional  knowledge  that  allows  them  to  generate  more  elaborations  than  other 
subjects.  Some  experiments  contrast  subjects  with  a  lot  of  domain-relevant  knowledge 
(for  example,  about  baseball),  to  those  with  little  relevant  knowledge  (e  g..  Chiesi.  Spilich 
&  Voss  1979;  Arkes  &  Freedman  1984  ).  Other  experiments  provide  some  of  the 

subjects  with  additional  information  that  is  relevant  to  a  passage  to  be  read.  For 

example,  Bower  (Bower,  Black  and  Turner  1979)  told  some  of  his  subjects  that  the 
passage's  protagonist  suspected  that  she  was  pregnant;  Sulin  and  Dooling  (1974)  told 
some  subjects  that  the  passage  was  about  Hitler  rather  than  Gerald  Martin.  Brown  et  al 
(1977)  instructed  half  of  the  subjects  about  the  fictitious  Targa  tribe  a  week  before  they 
read  about  it  in  the  stimulus  passage.  Subjects  who  have  more  relevant  knowledge  are 
more  likely  to  intrude  not-presented  but  relevant  information  and  are  also  more  likely  to 
false  alarm  to  plausible  inferences  based  on  this  additional  information.  Therefore  it  is 
reasonable  to  conclude  that  these  subjects  are  in  fact  elaborating  on  the  presented 

material  with  their  relevant  prior  knowledge.1  More  important  from  our  perspective,  these 
subjects  also  show  significantly  better  retention  for  the  gist  of  the  material  and  better 

understanding  for  it  (Anderson  &  Pichert  1977:  Arkes  &  Freedman  1984;  Bartlett  1932: 

Bower  1976.  Brown  et  al  1977;  Dooling  &  Cristiaansen  1977:  Owens  &  Bower  1977: 

Schallert  1976;  Weinstein  1978). 

Although  there  is  considerable  support  for  the  idea  that  reader-generated 
elaborations  facilitate  retention,  the  evidence  from  author-provided  elaborations  indicates 
that  they  are  not  as  beneficial  to  learning  as  expected.  Indeed,  most  of  the  research 

suggests  that  author-provided  elaborations  hurt  retention  of  the  central  ideas  .  Reder 

'pooling  a  Cnsiidansen  ,077V  for  example,  explored  to  what  extent  the  false  alarms  were  due  simply  to 
response  bias  as  opoosed  to  differential  encoding.  Although  part  of  the  effects  are  due  to  response  bias, 
part  of  the  result  is  clearly  due  to  encoding  differences. 
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and  Anderson  (1980,  1982)  conducted  ten  separate  experiments  on  author-provided 
elaborations  and  found,  to  their  surprise,  that  subjects  who  studied  fully-elaborated 
chapters  taken  verbatim  from  standard  college  textbooks  performed  consistently  worse  on 
recall  and  comprehension  tests  than  subjects  who  studied  summaries  that  were  one-fifth 
as  long.  No  matter  whether  the  retention  interval  was  20  minutes.  30  minutes,  one  week, 
or  6-12  months,  the  summary  condition  showed  a  consistent  advantage  on  true/false 
tests  and  on  short  answer  tests.  The  same  pattern  emerged  for  reaction  time 
differences  as  for  percentage  correct.  The  advantage  held  over  several  kinds  of  tests: 
Summaries  were  superior  for  questions  taken  directly  from  the  text  and  for  inference 
questions  that  required  the  subject  to  combine  studied  facts.  Allwood.  Wikstrom  & 
Reder  (1982)  found  the  same  advantage  for  summaries  in  a  free  recall  test,  in  a 
transfer  task,  subjects  learned  new  information  better  if  information  learned  earlier  on  a 
related  topic  was  acquired  by  reading  a  summary.  Mandl.  Schnotz  and  Tergan  (1984), 
who  also  found  an  advantage  for  summaries,  attributed  it  to  the  readers'  inability  to  pick 
out  the  important  points  in  the  elaborated  texts.  However,  in  a  study  designed  to 
facilitate  such  identification.  Reder  and  Anderson  (1980)  underlined  the  mam  points  in 
the  elaborated  texts,  but  the  summaries  still  maintained  their  advantage.  Reder  (1982) 
even  found  the  summary  advantage  to  hold  in  non-laboratory  situations  where  subjects 
took  the  material  home  to  study  for  as  long  as  they  desired.  The  pervasive  finding 
seems  to  be  that,  especially  with  a  fixed  study  time,  people  learn  facts  best  when  they 
study  those  facts  without  elaborative  information. 

Not  all  research  on  author-provided  elaborations  has  found  them  to  impair  learning. 
However,  the  conditions  under  which  such  elaborations  benefit  the  learner  tend  to  be 
rather  specialized.  For  example.  Stein  and  Bransford  (1979)  studied  subjects'  recall  of 
an  adjective  cued  bv  the  sentence  frame  within  which  it  had  ‘been  studied.  The 


elaborations  in  these  cases  were  additional  phrases  or  clauses  that  heightened  the 


importance  of  the  adjective  to  the  plausibility  of  the  sentence.  For  example,  subjects 

could  better  recall  the  adjective  "tall"  to  the  cue  "The  _  man  took  the  box  of 

crackers."  if  the  phrase,  "from  the  high  shelf"  had  also  been  studied. 

Bradshaw  and  Anderson  (in  press)  attempted  to  devise  especially  related 
elaborations  that  would  facilitate  recall  of  an  entire  sentence  (cued  by  the  sentence 
subject).  For  a  fixed  amount  of  study  time,  the  best  they  could  do  was  to  get  equivalent 
performance  in  the  elaborated  condition  with  learning  in  the  isolated  (unelaborated) 
condition. 

Mandl.  Schnotz  and  Tergan  (1984)  found  that  the  subject's  prior  knowledge 
interacts  with  the  usefulness  of  elaborations  Elaborated  texts  facilitated  recall  and 
comprehension  performance  only  if  the  reader  was  very  knowledgable  in  the  topic  area. 
For  less  knowledgeable  subjects,  the  elaborated  texts  produced  worse  performance  than 
unelaborated. 

Rothkopf  and  Billington  (1983)  found  that  elaborated  passages  were  sometimes 
superior  to  unelaborated  passages,  but  only  when  they  were  mixed  into  the  same  texts: 
when  subiects  read  one  text  that  was  uniformly  elaborated  and  another  that  was 
summarized,  the  summarized  version  produced  better  retention  of  the  central  points. 
Presumably,  the  former  result  for  the  mixed  passages  was  due  to  a  signalling  problem. 
That  is.  subjects  assumed  the  unelaborated  parts  of  the  text  were  less  important,  and 
therefore  subjects  paid  less  attention  to  the  shorter  segments.  Indeed.  Glover  has  found 


similar  results2 

The  finding  that  author-provided  elaborations  are  often  ineffectual  and  sometimes 
even  detrimental  to  learning  is  a  serious  and  curious  charge.  The  implications  for 
textbook  production  would  be  grave  if  one  actually  believed  the  result.  One  factor  that 
we  believe  is  crucial  to  the  effectiveness  of  the  elaborations  found  in  instructional  texts 
is  the  kind  of  learning  that  is  expected  to  take  place.  Indeed,  educators  and  laymen 
alike  will  often  assert  that  it  is  much  less  important  to  know  a  set  of  facts  than  to  know 
how  to  use  these  facts.  A  similar  distinction  between  fact  learning  and  skill  learning  has 
received  considerable  attention  in  recent  years  in  the  cognitive  science  literature  under 
the  label  of  declarative  versus  procedural  learning. 

We  believe  that  many  studies  found  elaborations  to  impair  learning  because  their 
tests  of  learning  demanded  only  the  retrieval  of  specific  facts  rather  than  the  application 
of  facts  to  a  skill.  Most  of  the  research  on  the  role  of  elaborations  in  memory  and 
learning  has  either  focused  exclusively  on  declarative  knowledge  or  it  has  ignored  the 
declarative/procedural  distinction.  As  a  resuit,  this  research  has  primarily  used  verification 
or  recall  as  measures  of  mastery,  tests  which  measure  the  ability  to  retrieve  information 
but  not  the  ability  to  "use"  the  information  in  the  sense  of  determining  the  appropriate 
contexts  of  application.  In  the  next  two  sections,  we  will  briefly  contrast  the  nature  of 
each  kind  of  learning  and  explain  why  we  expect  author-provided  elaborations  to  play  a 

^ln  other  unpublished  work  of  his,  Glover  and  his  colleagues  (Phifer,  McNickle.  Ronmng  &  Glover  1 9831 
found  that  summaries  produced  worse  performance  than  the  elaborated  texts  if  subjects  were  given 
approximately  one-fifth  the  reading  time  for  the  summaries  than  they  got  for  the  elaborated  text.  In  this 
experiment,  subjects  were  timed  reading  sentences  from  a  novel  and  then  had  to  read  the  summary 
sentences  at  the  same  average  rate.  Under  these  circumstances,  the  subjects  could  not  comprehend  the 
summaries,  and  their  performance  suffered.  Reder  and  Anderson  (1982)  used  a  different  method  of 
equating  reading  time  per  presented  sentence  for  the  elaborated  and  summary  texts,  but  with  a  slower 
overall  presentation  rate.  This  time  subjects  m  both  ronqitions  suent  me  same  amount  <>t  time  studying  the 
mam  points,  but  those  m  the  elaborated  condition  spent  additional  time  studying  the  elaborations.  Under 
these  conditions,  performance  was  much  better  without  the  "benefit”  of  the  elaborations. 


different  role  in  each. 


Why  do  author-provided  elaborations  impede  fact  learning? 

There  are  two  characteristics  of  tests  of  factual  knowledge  that  may  lead  a  person 
to  perform  better  after  studying  a  summary  than  a  full,  elaborated  text.  First,  tests  which 
ask  subjects  to  recall  or  recognize  studied  statements  require  retrieval  of  specific  facts 
learned  at  a  particular  time.  Some  of  these  tests  also  require  a  minimal  amount  of 
inferential  processing  of  the  retrieved  facts.  Good  performance  on  these  kinds  of  tasks 
results  from  strong  memory  traces  of  the  specific  facts  that  must  be  neved.  The 
stronger  the  trace  the  more  likely  that  it  will  be  retrieved  at  test.  A  pror  ion  or  fact  is 

strengthened  in  memory  to  the  extent  that  the  subject  devotes  moi  i  'otion  to  it. 

Studying  summaries  facilitates  performance  on  tests  of  factual  knowledge,  because  it 
allows  the  reader  to  devote  full  attention  to  the  essential  facts,  exactly  those  that  must 
be  retrieved  at  test. 

Under  this  analysis,  studying  elaborated  texts  impedes  learning  the  main  points  of 
the  text  because  reading  the  elaborations  reduces  the  amount  of  time  subjects  can 
devote  to  the  main  points.  The  Total  Time  Law  is  a  well  established  verbal  learning 

phenomenon  (e  g..  Bugelski  1962:  Cooper  &  Pantle  1967);  however,  it  can  not  completely 

explain  the  advantage  of  the  unelaborated  versions.  In  one  experiment  of  Reder  and 
Anderson  (1983).  study  time  was  equated  for  main  points,  i.e..  these  facts  were 
presented  for  equal  amounts  of  time  in  isolation  on  a  computer  terminal.  The  only 
difference  was  that  in  one  condition,  subjects  also  studied  additional  related  facts  after 
studying  each  main  idea.  Performance  was  significantly  worse  with  the  additional  facts. 
We  therefore  suggest  that  the  absence  of  interference  also  plays  a  significant  role  in  the 
superiority  of  the  summary  conditions. 
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Interference  differs  from  the  Total  Time  explanation  in  that  it  affects  retrieval  rather 
than  encoding.  There  is  ample  evidence  for  the  existence  of  retrieval  interference  on 
both  recall  (eg.,  Postman  1971:  Postman  &  Stark  1969)  and  on  response  times  to 
verification  (e.g.,  Anderson  1974:  Reder  &  Anderson  1980).  The  elaborations  not  only 
take  study  time  away  from  the  important  points,  they  also  interfere  with  accessing  these 
points  when  they  are  needed  later 


Another  reason  that  elaborations  do  not  help  performance  in  factual  tests  is  that 
there  is  little  uncertainty  in  the  testing  situation  about  how  to  apply  the  knowledge.  In  a 
recognition  or  a  recall  test,  it  is  usually  clear  what  information  is  needed,  when  or  why 
to  retrieve  it  and  what  to  do  with  it  once  it  is  retrieved.  The  information  found  in  an 
elaboration  is  seldom  called  on  in  such  tests  and  therefore  only  distracts  the  reader, 
when  his  or  her  time  would  be  better  spent  studying  the  targeted  facts  themselves  3 


In  short,  because  of  the  specific  demands  that  tests  of  fact  learning  place  on  the 
retrieval  processes,  studying  summaries  focuses  subjects'  attention  in  just  the  right 
places  to  produce  the  best  performance 


Why  might  author-provided  elaborations  facilitate  skill  learning? 

The  situation  in  tasks  requiring  skill  performance  is  quite  different.  The  application 
of  knowledge  to  a  task  of  skill  can  require  complex  judgments  of  the  appropriateness  of 
the  stored  information  to  the  context  --  judgments  that  are  not  required  in  the  simple 
memory  tests  that  have  heretofore  been  used  to  demonstrate  the  superiority  of 
summaries.  We  are  speaking  here  of  acquiring  a  cognitive  skill  that  the  learner  can 


JEssay  exams  do  not  fall  into  the  category  -J  .v.r  r  r-.  _  .  e  are  jefmmg  them.  Writing  an  essay 

dearly  calls  for  a  deeo  understanding  of  a  oodv  if  information  and  for  selecting  appropriate  items  from 
among  the  relevant  facts. 
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apply  to  a  variety  of  novel  situations.  This  type  of  skill  contrasts  quite  strongly  with  the 
kind  of  skill  required  to  follow  a  set  of  instructions  for  assembling  a  device  or  a  piece 
of  machinery.  The  skills  for  "assembly"  do  not  pose  problems  of  application.  Typically, 
there  is  usually  only  one  correct  way  to  follow  the  instructions  and  there  is  little  to 
memorize  since  one  usually  carries  out  the  instructions  immediately  after  reading  them 
On  the  other  hand,  the  cognitive  skills  we  are  investigating  (namely,  learning  to 
manipulate  files  and  directories  on  a  personal  computer),  can  be  used  much  more 
generally.  For  example,  the  command  for  copying  a  file  can  be  used  to  accomplish 
numerous  purposes,  and  the  most  correct  (or  most  efficient)  way  of  using  the  command 
depends  entirely  on  the  situation  at  hand.  Our  manual  teaches  the  general  procedure 
for  copying  files,  but  it  cannot  deai  explicitly  with  every  possible  variation  of  the 
command.  Furthermore,  there  may  be  more  than  one  command  or  combination  of 
commands  that  will  accomplish  the  user's  goal,  so  it  is  much  harder  to  determine  which 
procedure  should  be  applied  when. 

We  believe  that  studying  carefully  chosen  elaborations,  such  as  examples,  will  help 
a  learner  successfully  solve  novel  skill-related  problems.  Rather  than  distracting  the 
reader  away  from  the  main  points,  the  information  in  the  elaborations  may  make  it  easier 
to  decide  which  procedures  are  appropriate  to  the  situation.  For  example.  Ross  (1984) 
has  found  that  subjects  choose  between  equivalent  text-editing  procedures  on  the  basis 
of  the  similarity  between  the  nominal  situation  at  test  and  the  situation  evoked  in 
examples  found  in  the  materials  used  to  teach  the  procedures.  Further,  when  no 
studied  procedure  is  exactly  right  for  the  situation,  elaborations  can  make  it  easier  to 
modify  the  contexts  of  application  of  a  previously  learned  procedure  to  make  it  fit  the 
current  situation,  or  use  a  combination  of  separate  procedures. 

In  our  task  domain,  learning  to  use  a  personal  computer,  we  expect  examples  and 
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other  elaborations  to  play  an  important  role  for  helping  subjects  determine  how  to  issue 
specific  commands  to  the  computer,  as  well  as  deciding  when  a  particular  command  or 
strategy  is  appropriate.  Our  manuals  teach  subjects  how  to  manipulate  on-line  files  and 
directories  using  DOS.  the  Disk  Operating  System  on  the  IBM  Personal  Computer. 
During  the  criterion  test,  subjects  must  determine  which  command  or  commands  to  use 
to  achieve  a  set  of  specified  goals.  in  order  to  carry  out  their  plans,  they  must 

remember  the  exact  names  of  the  commands  they  want  to  use  as  well  as  the  specific 
syntax  of  the  commands,  (e  g.,  how  many  parameters  are  required  and  in  what  order 
they  appear).  Even  though  both  the  elaborated  and  unelaborated  versions  of  our 

manuals  contain  all  the  information  necessary  to  complete  all  of  the  tasks,  we  expect 
the  elaborated  version  to  help  subjects  complete  more  tasks,  more  efficiently.  We  expect 
examples  of  syntactically  correct  commands  to  help  subjects  remember  their  names  and 
syntax  more  easily,  and  to  help  them  induce  the  correct  commands  to  select  and  the 
most  efficient  manner  to  apply  them  to  a  task. 

In  summary,  we  suspect  that  the  findings  of  poorer  performance  for  elaborated 

tests  reflect  the  minimal  demands  that  tests  of  declarative  knowledge  make  on  the 

learner  s  understanding  of  the  text.  Although  elaborations  may  impair  learning  of  the 
main  points  of  a  text  by  competing  with  them  for  study  time  and  interfering  with  their 
retrieval,  we  believe  that  the  usefulness  of  elaborations  will  manifest  itself  in  more 
sophisicated  tests  of  knowledge.  Specifically,  we  expect  the  benefit  of  the  additional 
information  to  outweigh  the  liabilities  of  less  study  time  for  the  main  points  and  retrieval 
competition. 


The  effect  of  reader-generated  elaborations  on  skill-learning 


The  research  cited  earlier  on  reader-generated  elaborations  dealt  exclusively  with 
their  effects  on  declarative  learning.  Little  is  known  about  the  effect  of  reader-generated 
elaborations  on  skill  learning,  but  there  is  reason  to  believe  that  they  may  not  be  as 

trustworthy  as  those  the  author  can  provide.  When  the  reader  must  draw  inferences, 
invent  examples  and  anticipate  consequences  of  actions  in  an  unfamiliar  domain,  the 
resulting  elaborations  may  very  well  be  incomplete  or  inaccurate.  The  reader  may  be 
led  astray  by  prior  misconceptions  or  superficial  similarities.  On  the  other  hand,  the 

elaborations  a  reader  generates  are  more  likely  to  be  relevant  to  that  reader's  immediate 
goals.  To  the  extent  that  these  elaborations  are  more  easily  integrated  with  the  reader  s 
prior  knowledge,  they  may  also  be  more  memorable. 

In  an  attempt  to  control  the  degree  to  which  subjects  could  generate  effective 
elaborations,  we  provided  half  of  our  subjects  with  information  about  the  tasks  they 
would  have  to  perform  before  they  studied  the  user’s  manual  that  served  as  our 
instructional  text.  We  expected  that  as  these  subjects  read  the  manual,  they  would 
focus  on  those  parts  that  seemed  relevant  to  the  task,  generate  useful,  task-specific 
elaborations  and  ignore  anything  in  the  manual  that  did  not  seem  task-related  4 

We  expect  reader-generated  elaborations  to  improve  subjects'  performance,  but  it  is 
unclear  how  they  will  compare  to  author-provided  elaborations  and  how  they  will  interact. 
Are  subject-generated  elaborations  more  or  less  important  in  skill  learning  than  author- 
provided  elaborations?  Does  the  presence  of  one  source  of  elaboration  compensate  for 

the  lack  of  the  other?  Are  the  effects  additive  or  is  one  source  enough?  Do  they 


^Providing  qnor  knowledge  of  (he  test  material  has  been  explored  extensively  in  test  of  declarative 
memory.  See  R.C.  Anderson  &  Biddle  (1975).  Reder  (t982).  fleder  <m  press)  and  Rickards  <  ? 979)  for 
reviews. 


interfere  with  each  other? 


Experiment  1 

In  this  experiment,  subjects  were  asked  to  read  one  of  two  versions  of  a  user  s 
manual  (that  we  wrote)  on  how  to  use  the  IBM  Personal  Computer  (IBM-PC).  We  then 
measured  their  facility  at  using  an  IBM-PC  without  the  manual  when  they  were  given 
specific  tasks  to  perform  on  the  machine. 


Method 


Subjects 


Forty-five  members  of  the  Carnegie-Mellon  University  community,  33  students.  3 
faculty,  and  9  staff  members  participated  in  this  experiment.  All  subjects  were 
occasional  to  frequent  users  of  C-MU'S  DEC-20  computer  system.  Three  subjects  were 
non-native  speakers  of  English,  but  fluent  enough  to  completely  understand  the 
documentation.  Subjects  received  either  money  or  class  participation  credit  or  a 
combination  of  the  two.  A  subject  could  receive  either  $8.00,  $4.00  plus  one  credit,  or 
2  credits  (out  of  a  required  3  credits)  for  participating. 


Design 


The  experiment  used  a  2x2  between-subjects  factorial  design,  where  the  first 
variable  manipulated  whether  the  document  contained  elaborations  or  not.  and  the 
second  varied  whether  subjects  read  the  task  instructions  prior  to  studying  the  document 
or  not.  Subjects  were  randomly  assigned  to  conditions,  with  the  constraint  that  ten 
subjects  be  assigned  to  each  of  the  four  conditions.  Although  we  noted  how  much 
previous  computer  experience  our  subjects  had.  we  did  not  control  for  this  variable  in 
assignment  to  conditions,  except  for  ensuring  that  no  subject  had  ever  worked  on  a 
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microcomputer.  Rather,  we  used  prior  experience  as  a  covariate  in  our  data  analyses. 

Performance  was  measured  with  both  a  procedural  test  (actually  asking  the 
subjects  to  do  something  on  the  computer)  and  a  surprise  declarative  test  (completing  a 
paper  and  pencil  questionnaire).  These  measures  are  described  in  detail  in  the  Results 
section. 

Materials 

Documents.  Two  versions  of  a  user’s  manual  were  developed  for  teaching  novices 
to  use  the  DOS  operating  system  on  an  IBM  personal  computer.  Both  versions  were 
constructed  by  modifying  the  official  documentation  published  by  IBM.  The  same  main 
points  were  presented  in  both  versions;  they  differed  only  in  the  degree  of  elaboration  of 
the  conceptual  information  and  the  commands  described.  The  manuals  were  divided  into 
two  sections.  The  first  section  discussed  concepts  underlying  the  IBM-PC  and  its 
operating  system.  These  topics  included  disk  drives,  directories  and  subdirectories,  and 
the  use  of  wildcard  characters.  This  section  also  contained  general  information  about 
how  to  use  the  machine,  how  to  use'  the  manual,  how  to  issue  commands  to  the 
machine,  etc.  The  first  section  was  intended  to  prepare  the  subject  to  give  the 
commands  specified  in  the  second  section  of  the  document,  since  issuing  commands 
often  requires  specifying  the  location  of  a  file  in  terms  of  the  disk  drive  the  file  is  on. 
and  a  path  to  the  file  through  a  network  of  directories  and  subdirectories. 

The  second  section  of  the  documentation  contained  information  about  the  eleven 
commands  that  we  taught  our  subjects.  A  complete  list  of  these  commands  is  provided 
in  the  Appendix.  When  describing  or  introducing  a  command,  the  manual  explains  what 
the  command  does,  what  parameters  must  be  specified  when  the  command  is  issued, 
any  optional  parameters,  and  any  other  special  information  about  it.  The  descriptions  of 
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the  commands  also  contain  options  for  modifying  the  commands,  if  any,  and  any  special 
information  about  that  command.  For  example,  the  section  on  the  DIR  command 
describes  an  option  for  producing  a  more  compact  display,  and  an  option  for  making  the 
display  pause  when  the  computer  screen  is  full.  Several  sections  include  information  on 
how  to  use  wildcards  with  particular  commands.  Table  i  gives  examples  of 

documentation  from  the  first  section  and  from  the  second  section  for  both  versions  of 
the  documentation. 

The  unelaborated  version  of  the  document  was  constructed  by  deleting  portions  of 
the  elaborated  manual,  such  as  examples,  analogies,  metastatements,  and  definitions. 
Table  2  gives  samples  of  each  of  these  types  of  elaboration  from  the  elaborated 
manual,  with  the  corresponding  sections  of  the  unelaborated  manual  printed  alongside. 
The  elaborations  did  not  contain  any  information  that  would  be  needed  to  successfully 
complete  the  tasks  we  asked  subjects  to  perform.5  The  elaborated  version  contained 
approximately  11,200  words,  while  the  unelaborated  version  contained  approximately  5.000 
words. 


5Cleariv  the  specific  nature  of  the  elaborations  piavs  a  role  m  how  useful  they  are  to  learning  and  to 
performance  on  specific  'asks,  no-e'.er.  -or  --e  -lesent.  «e  are  relying  on  intuition  to  help  us  select  the 
most  useful  elaborations  we  can.  primarily  examples  ana  illustrations.  We  will  discuss  later  the  issue  of  how 
different  types  of  elaborations  might  manifest  differents  effects  on  subiect  behavior. 


Table  1 


Excerpts  from  the  elaborated  and  unelaborated 
versions  of  the  manual  used  in  Experiment  1 


Section  1:  Basic  concepts 


ELABORATED 

GIVING  THE  COMPUTER  A  COMMAND 

In  order  for  the  computer  to  store 
information  or  perform  a  calculation, 
you  have  to  give  it  a  command 
which  specifies  exactly  what  you  want 
it  to  do.  To  give  a  command,  you  type 
words  and  symbols  in  a  fixed  format 
that  the  computer  recognizes.  The 
commands  you  will  give  the 
computer  consist  of  three  parts: 
the  name  of  the  command,  the 
command's  ’parameters,"  and  a 
carriage  return  <CR>.  The  computer 
'prompts'  you  when  it  is  ready  for  you 
to  type  a  command  by  displaying 
either  A>  or  B>  on  the  screen. 
Throughout  this  book  this  "prompt" 
is  highlighted  so  you  will  remember 
that  the  computer  types  this, 
not  you. 

For  example,  the  following  command 
tells  the  computer  to  find  a  file 
called  REPORT.MSS,  and  type  the 
contents  of  this  file  on  the  compuer 
screen: 

A>TYPE  REPORT.MSS  <CR> 

CMD  NAME.  The  name  of  the  command 
is  usually  an  English  word  or 
abbreviation,  such  as  TYPE  in  the 
example  above.  The  computer  uses  the 
name  of  the  command  to  look  up  a 
program,  a  set  of  instructions  for 
carrying  out  the  command.  The 
commands  that  the  computer  recognizes 
are  discussed  individually  later  in 
this  book.  The  computer  recognizes 
some  abbreviations  of  the  command 
names,  such  as  REN  for  RENAME,  and 
the  accepted  abbreviations  will  be 
listed  with  the  description  of  each 
command. 


UNELABORATED 

GIVING  THE  COMPUTER  A  COMMAND 

In  order  for  the  computer  to  store 
information  or  perform  a  calculation, 
you  have  to  give  it  a  command 
which  specifies  exactly  what  you  want 
it  to  do.  To  give  a  command,  you  type 
words  and  symbols  in  a  fixed  format 
that  the  computer  recognizes.  The 
commands  you  will  give  the 
computer  consist  of  three  parts: 
the  name  of  the  command,  the 
command's  ’parameters,"  and  a 
carriage  return  <CR>.  The  computer 
"prompts’  you  when  it  is  ready  for  you 
to  type  a  command  by  displaying 
either  A>  or  B>  on  the  screen. 
Throughout  this  book  this  "prompt" 
is  highlighted  so  you  will  remember 
that  the  computer  types  this, 
not  you. 

The  general  format  of  a  command 
statement  looks  like  this: 


A>  [CMD  NAME]  [PARAMETER]  <CR> 

CMD  NAME.  The  name  of  the  command 
is  usually  an  English  word  or 
abbreviation.  The  commands  that  the 
computer  recognizes  are  discussed 
individually  later  in  this  book, 
along  with  acceptable  abbreviations 
for  each  one. 
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Table  1,  continued 


Section  2:  Specific  commands 

ELABORATED  UNELABORATED 


CHANGE  THE  NAME  OF  A  FILE:  RENAME 


CHANGE  THE  NAME  OF  A  FILE:  RENAME 


Use  the  RENAME  command  to  change 
the  name  of  a  file.  You  may  want  to 
change  a  file's  name  to  make  it 
easier  to  remember  or  to  make  it 
similar  to  the  names  of  other  files 
that  contain  related  information.  You 
can  use  RENAME  to  change  the  stem  part 
of  a  filename,  its  extension,  or  both. 


Use  the  RENAME  command  to  change 
the  name  of  a  file. 


FORMAT 


FORMAT 


The  format  of  the  command  is:  The  format  of  the  command  is: 

RENAME  [loc  &  name  of  file  to  be  renamed]  [new  name  of  file] 


You  can  use  the  abbreviation  REN  in 
the  command  instead  of  typing 
RENAME. 


You  can  use  the  abbreviation  REN  in 
the  command  instead  of  typing 
RENAME. 


[Loc  &  name  of  file  to  be  renamed] 
refers  to  the  path  to  the  file  you 
want  to  rename.  Type  the  name  of  the 
file  as  the  last  name  on  the  path. 


[Loc  &  name  of  file  to  be  renamed] 
refers  to  the  path  to  the  file  you 
want  to  rename.  Type  the  name  of  the 
file  as  the  last  name  on  the  path. 


[New  name  of  file]  is  what  you  want 
to  call  the  file  from  now  on.  You 
simply  type  the  new  filename.  You  do 
not  have  to  supply  any  location 
information,  since  the  renamed  file 
stays  in  the  same  place  it  was  before. 


[New  name  of  file]  is  what  you  want 
to  call  the  file  from  now  on.  You 
simply  type  the  new  filename.  You  do 
not  have  to  supply  any  location 
information,  since  the  renamed  file 
stays  in  the  same  place  it  was  before. 


For  example,  the  following  RENAME 
command  specifies  that  the  file 
BUDGET  should  be  renamed  BUDGET.83: 


A>  RENAME  8:8UDGET  BUDGET.83  <CR> 


The  path  indicates  that  BUDGET  is 
located  in  the  current  directory  on 
drive  B.  The  new  name  for  the  file, 
BUDGET.83,  is  not  preceded  by  any 
path  information. 


1-  -  ’  - 1- 


Table  2 


Samples  of  four  types  off  elaboration 
from  the  manual  used  in  Experiment  1 


Meta-statenment 


ELABORATED 

Since  the  computer  has  two  disk  drives 
which  can  each  contain  a  diskette,  you 
must  specify  whether  the  file  you  want 
is  in  drive  A  or  drive  B  when  you 
give  the  computer  a  command.  If  your 
command  doesn't  specify  which  drive 
contains  the  file,  the  computer 
automatically  assumes  that  it  can  find 
the  file  in  the  ''default'  drive.  The 
next  section  explains  what  the  'default 
drive*  is,  and  how  to  teil  the  computer 
to  look  on  a  different  drive  if 
necessary. 


UNELABORATED 

Since  the  computer  has  two  disk 
drives  which  can  each  contain  a 
diskette,  you  must  specify 
whether  the  file  you  want  is  in  drive 
A  or  drive  B  when  you  give  the 
computer  a  command.  If  your 
command  doesn't  specify  which 
drive  contains  the  file,  the 
computer  automatically  assumes 
that  it  can  find  the  file  in 
the  "default'  drive. 


Definition) 


ELABORATED 

The  B:  ('B-colon')  in  the  command 
stands  for  the  right-hand  disk  drive. 
The  colon  signals  the  computer  that 
the  letter  or  word  preceding  it  is  a 
'device'  rather  than  the  name  of  a 
command  or  file.  Devices  are  pieces 
of  computer  hardware,  such  as  disk 
drives,  a  printer  or  even  the  keyboard. 
After  you  enter  the  command,  the  B> 
prompt  will  appear  on  the  screen. 
From  now  on  the  computer  will 
automatically  look  for  files  on  drive  B. 


UNELABORATED 

The  B:  in  the  command  stands  for 
the  right  hand  disk  drive.  From 
now  on,  the  B>  prompt  appears  on 
the  screen  and  the  computer  will 
automatically  look  for  files  on 
drive  B. 


Table  2.  continued 


Analogy 


ELABORATED 

When  you  give  the  computer  a  command 
concerning  a  file,  such  as  TYPE,  ERASE 
or  COPY,  the  computer  looks  for  the 
file  on  a  "diskette.*  A  diskette,  also 
known  as  a  'floppy  disk,*  is  similar 
to  a  small,  flexible  phonograph  record 
record,  except  that  instead  of  storing 
sounds,  it  contains  information  which 
the  computer  can  read,  add  to  or  delete. 
All  the  files  you  create  on  the  computer 
are  stored  on  diskettes.  So,  in  order 
to  work  on  your  files,  you  must  insert 
the  diskette  that  contains  them  into 
the  computer.  You  insert  a  diskette 
into  one  of  the  two  'disk  drives*  on 
the  front  of  the  computer  cabinet.  The 
drive  on  the  left  is  called  drive  A, 
and  the  one  on  the  right  is  drive  B. 


UNELABORATED 

When  you  give  the  computer  a 
command  concerning  a  file,  such  as 
TYPE,  ERASE  or  COPY,  the  computer 
looks  for  the  file  on  a  'diskette.* 

To  use  a  diskette,  you  insert  it  into 
one  of  the  two  'disk  drives*  on  the 
front  of  the  computer  cabinet. 

The  drive  on  the  left  is  called 
drive  A,  and  the  one  on  the  right 
is  drive  B. 
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Table  2.  continued 


Example 


The  elaborated  and  unelaborated  versions 
are  for  the  most  part  identical  except  for 
the  addition  of  the  example  (italicized  here) 


Using  COPY  to  Combine  Files 

You  can  use  COPY  to  combine  files,  appending  a  copy  of  one  file  to  the  end 
of  another  file. 

FORMAT 


The  format  of  the  command  is: 


COPY  [Loc  &  name  first  file+next  file*...]  [Loc  &  name  combined  file] 

[Loc  &  name  first  file+next  file...]  refers  to  a  list  of  the  files  you  want 
to  'add'  together.  The  names  of  the  files  are  typed  with  plus  (+)  signs 
between  them.  You  need  to  specify  location  information  for  each  filename  in 
the  list  in  the  usual  way,  with  drive  and  path  specifications.  When  several 
filenames  are  listed  in  this  manner,  the  COPY  command  results  in  a  new  file 
in  which  the  contents  of  the  first  file  on  the  list  appear  first,  followed  by 
the  contents  of  the  second  file,  then  the  contents  of  the  third  file  and  so 
on.  So  be  sure  that  the  files  in  the  list  appear  in  the  order  in  which  you 
want  them  combined. 

[Combined  file]  refers  to  the  new  file  that  will  contain  the  combined  files; 
what  you  want  to  call  this  file,  and  where  in  the  directory  structure  you 
want  it  to  go.  Specify  the  location  in  terms  of  a  drive  and  a  path  to  a 
directory  as  usual.  Type^the  naraa  you  would  like  to  give  the  file  at  the 
end  of  the  path. 

For  example,  suppose  you  write  a  report  in  sections,  with  each  section  in  a 
separate  file.  You  want  to  format  and  print  the  report  as  one  file,  so  you 
combine  the  sections  into  one  file.  The  following  command  takes  three 
files,  INTRO.MSS,  BOOY.MSS,  and  CONCL.MSS  and  combines  them  into  a  new  file 
called  REPORT: 


A>  COPY  B:INTRO.MSS+B:BODY.MSS+B:CONCL.MSS  REPORT  < ENTER > 

The  combined  file,  REPORT  will  consist  of  the  introduction,  the  body  and  the 
conclusion. 
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Test  materials.  Two  tests  were  designed  to  cover  the  material  presented  in  the 
manual:  one  procedural,  the  other  declarative.  The  procedural  task  tested  subjects' 
ability  to  use  8  of  the  1 1  commands  on  line,  from  memory.  This  task  will  be  described 
in  more  detail  in  the  procedure  section.  The  paper-and-pencil  declarative  test  covered  all 
1 1  commands,  including  some  special  uses  of  the  same  commands  that  were  required  in 
the  procedural  task.  Half  of  the  declarative  knowledge  questions  required  the  subiect  to 
write  a  command  that  would  produce  a  specified  result  if  issued  on  the  computer.  The 
other  half  of  the  questions  required  that  the  subject  describe  (in  writing)  what  effect  a 
specified  command  would  have  in  a  specified  context.  The  commands  taught  in  the 
manual  were  randomly  assigned  to  either  the  "generate  command"  or  the  "describe 
result"  question  type.  An  example  of  each  type  of  question  is  provided  in  the  Appendix. 
There  were  1 1  questions  in  all. 

Apparatus 

Two  IBM  Personal  Computers,  each  with  two  floppy  disk  drives,  were  used  in 
running  the  experiment.  The  PCs  stood  on  nearby  tables  and  were  connected  together 
by  a  cable  between  their  serial  ports.  Software  was  developed  to  collect  protocols  of  the 
subiect  s  interactions  with  the  computer.  While  the  subject  sat  and  issued  commands  at 
one  PC.  the  commands  and  the  computer  s  responses  were  echoed  across  the  cable 
and  recorded  (with  a  timestamp)  in  a  file  on  the  second  PC.  A  room-divider  blocked 
the  subject's  view  of  the  second  PC's  screen. 

Procedure 

All  subjects  saw  the  following  instructions,  either  before  or  after  studying  the  user’s 


manual. 


Before  you.  in  drive  A.  is  a  diskette  containing  a  number  of  files.  These  files 
have  arbitrary  meaningless  names,  such  as  "FI."  We  want  you  to  inspect 
each  file  and  rename  it  according  to  the  name  listed  in  the  first  line  of  the 
file. 

After  renaming  each  file,  we  would  like  you  to  create  a  number  of 
subdirectories.  Name  each  sub-directory  according  to  the  files  that  will  be 
stored  under  it.  You  should  sort  the  files  into  subdirectories  on  the  basis  of 
their  ''stem''  names.  That  is.  if  you  had  a  file  named  ’'foo.pas'’.  you  would 
sort  it  by  "foo."  not  by  "pas'.  Pit  all  files  with  the  same  stem  into  a  sub¬ 
directory  bearing  that  name. 

If  there  are  files  that  have  a  unique  stem  name.  i.e..  no  other  file  has  the 
same  stem  name,  then  we  want  you  to  "migrate"  that  file  from  the  current 
diskette.  To  migrate  the  files,  you  will  copy  them  to  the  diskette  on  drive  B. 
and  then  delete  them  from  the  diskette  on  drive  A.  You  should  also  create  a 
file  called  "migrate. 1st"  at  the  top  level  directory  of  the  diskette  in  Drive 
A.  Type  the  names  of  afl  the  files  you  migrated  into  migrate. 1st. 

To  summarize  your  tasks: 

1)  Inspect  each  file  to  discover  its  intended  name.  Rename  each  file 
according  to  the  name  found  on  the  first  line  in  that  file. 

2)  Create  a  hierarchical  set  of  directories  corresponding  to  the  types  of  files 
that  you  have.  Create  a  sub-directory  for  each  "type"  of  file,  where  type  is 
defined  in  terms  of  the  communality  of  stem  names.  Transfer  the  appropriate 
files  into  each  subdirectory. 

3)  For  those  files  that  have  a  unique  stem  name,  do  not  create  a  sub¬ 
directory.  instead  copy  those  files  to  the  diskette  on  Drive  B.  and  delete  them 
from  the  directory  on  Drive  A. 

4)  At  the  top  directory  level  on  the  diskette  in  Drive  A.  you  should  now 
have  a  set  of  subdirectories,  but  no  files.  You  should  create  a  file  in  this 
directory  that  lists  all  the  files  that  were  copied  to  the  other  diskette. 

Now.  you  are  done  with  the  tasks! 


Those  subjects  who  were  randomly  assigned  to  the  condition  that  allowed  them  to 
learn  the  specifics  of  the  task  before  reading  the  manual  (the  Before  condition)  read  the 
task  instructions  before  being  given  their  version  of  the  manual.  The  other  half  of  the 
subjects  (the  After  condition)  read  through  the  task  instructions  for  the  first  time  when 
they  were  about  to  perform  the  tasK.  naif  of  each  of  these  groups  were  given  the 


elaborated  version  to  read  and  half  the  shorter  version. 

All  subjects,  regardless  of  condition,  studied  the  manual  for  45  minutes.  They 

were  told  that  since  the  manual  would  not  be  available  while  they  performed  the  task, 

they  should  review  the  important  points  of  the  manual  as  often  as  time  permitted. 
Subjects  were  told  when  the  reading  period  was  half  over.  Subjects  studied  the  manual 
while  seated  in  front  of  the  computer  they  were  reading  about.  They  were  told  to 
examine  the  keys  referred  to  in  the  documentation:  however,  they  were  also  instructed 

not  to  issue  any  commands  or  otherwise  touch  the  machine,  and  not  to  take  notes. 

When  the  reading  period  was  over,  the  experimenter  returned  to  the  room  where  the 

subject  studied  the  document  to  give  the  subject  the  task  instructions  (for  the  second 
time  if  the  subject  was  in  the  Before  condition). 

Even  though  the  computer  recorded  all  interactions  with  the  subject,  the 
experimenter  remained  in  the  room  while  the  subject  attempted  to  perform  the  specified 
tasks.  It  was  the  responsibility  of  the  experimenter  to  determine  if  the  subject  had 

arrived  at  an  impasse,  i.e..  could  not  complete  part  or  all  of  the  current  task.  Either 
the  subject  gave  up  or  was  stopped  after  approximately  10  minutes  of  fruitless  effort. 
At  this  point,  the  experimenter  replaced  the  diskette  the  subject  was  working  on  with  a 
prepared  diskette  on  which  the  procedures  for  that  task  or  sub-task  had  already  been 
completed,  in  this  way,  the  subject  could  proceed  to  the  next  task  as  if  he  or  she  had 

actually  completed  the  sub-task.  The  experimenter  also  took  notes  on  any  occurrences 

that  would  not  be  recorded  by  the  computer  (such  as  spoken  comments).  Subjects  were 
not  allowed  to  ask  questions,  unless  the  question  was  of  a  superficial  nature,  e.g.,  which 
key  was  the  carriage  return.  (This  was  not  obvious  on  the  IBM-PC  -keyboard). 

After  all  the  requirements  of  the  task  instructions  were  completed  to  the  subject  s 
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satisfaction  or  abilities,  the  surprise  declarative  (short-answer)  test  was  administered. 
Subjects  were  allowed  to  take  as  long  as  they  wanted  to  complete  this  test;  however 
they  generally  took  about  15  minutes.  The  entire  experiment  took  approximately  1.5  to 
2  hours,  depending  on  the  subject's  ability. 

RESULTS 

The  following  results  reflect  the  scores  of  40  subjects.  In  all,  45  subjects 

completed  the  study,  but  the  data  from  five  subjects  were  either  thrown  out  or  were 

unusable.  Data  from  three  subjects  were  lost  because  of  computer  failure.  Data  from 
two  subjects  were  thrown  out  because  they  had  so  little  previous  computer  experience 
that  they  refused  to  continue  the  experiment  shortly  after  beginning  the  procedural  task. 

Scoring 

The  protocol  of  a  subject's  interactions  with  the  computer  were  stored  in  a  file 

which  was  subsequently  analyzed  by  means  of  a  computer  program.  The  program 

allowed  the  experimenter  to  categorize  segments  of  the  protocol  according  to  the 
subtask  the  subject  was  working  on.  The  subjects  task  can  be  divided  into  three 
separate  sub-tasks: 

1.  Rename  task;  inspect  the  contents  of  all  files  on  diskette  and  rename  them. 

2.  Make-directory  task:  create  three  subdirectories  and  move  the  appropriate  files 

into  them  from  the  root  directory. 

3.  Migrate  task:  ship  files  from  one  disk  drive  to  another  and  create  a  new  file. 

The  program  produced  a  variety  of  statistics  about  each  protocol  by  counting 
commands  and  calculating  time  intervals,  either  within  or  across  subtask  "partitions." 
For  example,  the  program  could  determine  how  many  times  a  subject  had  issued  a 
"TYPE"  command  while  working  on  the  Rename  task,  and  how  much  time  (in  seconds) 


it  took  him  or  her  to  issue  those  commands.  The  partitioning  of  protocols  into  "subtask 
partitions"  was  carried  out  independently  by  two  judges  for  a  random  subset  of  the 
protocols.  The  agreement  between  the  judges  was  quite  high  (the  regression  coefficients 
ranged  from  r  =  .98  to  r=l.O),  and  any  disagreements  were  resolved  to  mutual 

satisfaction.  (The  appendix  contains  an  excerpt  from  a  subject's  protocol  before  and 
after  it  was  analyzed  using  this  program). 

The  data  were  scored  in  several  ways.  One  dependent  measure  was  simply  the 
total  time  a  subject  spent  attempting  to  complete  the  task  requirements.  Another 
measure  was  total  steps,  or  the  number  of  commands  the  subject  issued  while  working 
on  the  task.  A  third  measure  was  a  completeness  measure,  viz.  how  many  of  the  sub¬ 
tasks  were  successfully  completed.  Because  of  the  large  variation  in  times  to  "quit"  on 
a  sub-task,  we  also  measured  how  much  time  subjects  took  on  just  those  sub-tasks  they 
successfully  completed.  Since  the  sub-tasks  themselves  varied  in  the  average  time 

needed  for  successful  completion,  we  looked  at  how  a  subject’s  time  on  a  successfully 
completed  task  deviated  from  the  mean  of  times;  for  all  successful  completions  on  that 
task.  Then,  for  each  subject,  we  calculated  a  mean  of  proportional  times  for  the  tasks 
he  or  she  completed.  Similarly,  we  computed  a  subject’s  mean  of  proportional  steps 
(i.e. .  number  of  steps/avg.  number  of  steps  for  all  subjects  who  completed  this  task)  for 
each  completed  sub-task.  Finally,  we  compared  the  number  of  steps  subjects  took  on 
tasks  they  completed  with  the  minimum  number  of  steps  required  to  complete  those 
tasks. 

We  developed  a  completeness  scale  to  score  subjects  for  how  successfully  they 
completed  each  of  the  three  sub-tasks.  In  order  for  the  Rename  task  to  be  complete, 
each  of  the  15  files  must  have  been  correctly  renamed.  If  a  subject  correctly  renamed 

some  but  not  all  of  the  files  (which  never  happened),  or  used  the  "copy"  command 
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(which  creates  an  extra  copy  of  a  file),  instead  of  the  "rename"  command  and  then 
failed  to  delete  the  version  with  the  old  name,  the  subject  received  a  score  of  one-half 
instead  of  one  point. 

In  the  Make-directory  sub-task,  subjects  received  3  points  if  all  three  sub-directories 
were  created  and  all  the  appropriate  files  were  copied  into  them  and  then  deleted  from 
the  root  directory.  For  any  subdirectory  that  was  not  properly  created  or  did  not  contain 
all  the  files  it  should,  subjects  lost  a  point. 

In  the  Migrate  sub-task,  one  file  had  to  be  copied  to'  the  diskette  on  drive  B. 
erased  from  drive  A,  and  a  new  file  created  which  listed  the  name  of  that  file.  The  last 
step  involved  a  special  form  of  the  COPY  command  which  enabled  the  subject  to  copy 
data  typed  at  the  console  keyboard  into  a  file.  If  each  of  these  three  steps  was 

executed,  the  subject  received  three  points. 

Performance  on  Task 

Success  on  Tasks 

The  principal  dependent  measure  is  the  degree  to  which  a  subject  was  able  to 
satisfactorily  perform  the  various  tasks  specified  in  the  instructions.  Table  3  shows  the 
proportion  of  tasks  completed  as  a  function  of  version  of  document  and  whether  the 

instructions  were  read  before  the  manual  or  only  after.  There  is  a  2%  advantage  for 

reading  the  elaborated  version  and  a  5%  advantage  for  not  seeing  the  task  instructions 
until  after  studying  the  manual:  however  neither  of  these  effects  is  significant.  When 
subjects  did  not  know  what  tasks  to  expect,  the  difference  between  the  two  versions  of 
the  manual  is  almost  20%:  however,  in  the  Before  condition,  there  is  no  difference. 

Although  the  interaction  is  not  significant,  the  superior  performance  of  the  Elaborated- 
After  condition  compared  to  the  other  three  conditions  is  significant.  t(38)  =  2,98,  p<  .05. 


This  condition  is  about  15%  better  than  any  of  the  other  three,  and  these  three  are 
about  equivalent  in  overall  performance.  When  broken  down  by  subtask,  this  measure 
produces  the  same  pattern  for  all  three  subtasks,  as  shown  in  Table  4.  The  rename  task 
is  so  easy  that  performance  is  on  the  ceiling;  however,  for  the  second  and  third  tasks 
(creating  new  directories  and  moving  files  to  another  drive),  performance  is  much  better 
for  the  Elaborated-After  condition.  t(38)  =  1 . 98  and  2.62.  respectively.  The  former  contrast 
is  marginally  significant  and  the  latter  is  significant.  p<.05. 


TABLE  3 
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Mean  proportion  of  tasks  successfully  completed 
for  Experiment  1* 


VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before 

1 - 

1 

- r 

.79  1 

.79 

— r 

1  .79 

1 

1 

(.80)  1 

1 

(.80) 

1  (.80 

1 

After 

1 

1 

.93  1 

.75 

1  .84 

1 

1 

(.92)  1 

1 

(.82) 

1  (.87 

1 

.86 

(.86) 


.77 

(.81) 


The  numbers  in  parentheses  are  the  means  for  each  cell 
adjusted  for  the  number  of  prior  programming  languages.  See 
text  for  further  explanation. 
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Efficiency  of  task  execution:  number  of  steps 

In  addition  to  measuring  a  subject’s  ability  to  perform  a  task,  one  can  ask  whether 
a  subject  performed  the  task  in  the  optimal  manner.  For  example,  we  asked  subjects  to 
change  the  names  of  a  set  of  files.  A  name  change  can  be  accomplished  by  using  the 
COPY  command  to  create  an  additional  version  of  the  file  with  a  new  name,  and  then 
using  the  ERASE  command  to  delete  the  version  of  the  file  with  the  old  name. 
However,  this  method  takes  more  steps  to  complete  than  using  the  RENAME  command, 
which  does  not  involve  creating  another  version  of  the  file.  Our  scoring  procedure 
allowed  us  to  tally  the  number  of  steps  a  subject  took  (i.e..  how  many  commands  he  or 
she  issued)  within  each  subtask.  The  mean  number  of  steps  subjects  took  to  perform 
all  tasks  is  presented  as  a  function  of  condition  in  Table  5a.  Here,  as  with  the 
completeness  measure,  the  elaborated  manual  produces  the  best  performance,  and 
reading  the  manual  without  advance  knowledge  of  the  tasks  also  improves  performance. 
As  expected,  due  to  the  high  variance  in  the  time  unsuccessful  subjects  took  to  "quit" 
on  a  task,  none  of  the  contrasts  are  reliable. 

The  replacement  for  the  Total  Steps  measure  avoids  the  problem  of  artificially  high 
numbers  of  steps  (when  subjects  persist  in  following  some  dead-end  path),  and  artificially 
low  numbers  (when  subjects  quickly  realize  that  they  cannot  complete  the  task  and  give 
up  right  away).  We  did  not  use  the  mean  number  of  steps  taken  per  successfully 
completed  subtask  because  the  number  of  steps  required  for  a  subtask  is  not  standard. 
The  mean  for  a  subject  could  be  biased  high  or  low  depending  on  which  subtasks  were 
completed.  Instead,  we  computed  the  mean  of  the  number  of  steps  needed  by  those 
subjects  who  successfully  completed  each  subtask.  A  subject's  score  on  a  subtask 
became  the  ratio  of  the  number  of  steps  he  or  she  took  divided  by  the  mean  number 
of  steps  taken  to  successfully  complete  that  task.  So.  to  arrive  at  an  overall  score  for 


each  subject,  we  averaged  the  proportional  scores  for  those  tasks  the  subject 
successfully  completed. 

Table  5b  presents  the  mean  proportion  of  steps  taken  per  completed  task.  A 
number  close  to  unity  should  be  considered  baseline.  As  this  table  indicates,  the  two 
Before  conditions  produce  normative  performance,  while  performance  in  the  After 
conditions  varies  with  the  degree  of  elaboration  in  the  manual:  the  Elaborated-After 
condition  produces  better  than  average  performance  and  the  Unelaborated-After  condition 
produces  worse  than  average  performance.  Both  the  main  effect  of  elaboration  and  the 
interaction  of  elaboration  with  prior  task  knowledge  are  marginally  significant. 
F(1,35)  =  3.08  and  2.91,  pc. 10,  respectively.  The  Elaborated-After  condition  maintains  its 
status  as  the  best  condition:  the  contrast  of  Elaborated-After  to  the  other  three 
conditions  is  significant.  t(38)=  -2.65.  p<  .05. 

In  addition  to  comparing  subjects’  efficiency  relative  to  each  other,  we  also 
compared  the  number  of  steps  each  subject  took  to  complete  a  subtask  against  the 
minimum  number  of  steps  required  li  e.,  the  most  efficient  solution)  to  complete  that 
subtask.  For  this  measure,  a  subject  s  score  on  a  subtask  was  the  ratio  of  the  number 
of  steps  he  or  she  took  divided  by  the  minimum  number  of  steps  required.  To  arrive  at 
an  overall  score  for  each  subject,  we  averaged  the  proportional  scores  for  those  tasks 
the  subject  successfully  completed. 

Table  5c  presents  the  mean  efficiency  ratio  per  completed  task.  The  greater  the 
value,  the  less  efficient  the  performance.  As  the  table  indicates,  performance  is  again 
best  in  the  Elaborated-After  condition  and  worst  in  the  Unelaborated-Before  condition,  but 


no  effects  were  significant. 


Table  5a 


Mean  number  of  steps  taken  by  subjects 
to  perform  all  subtasks 
for  Experiment  1 


Before 
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TABLE  5b 


Mean  proportion  of  steps  taken  per  successfully  completed  task 
(compared  with  the  mean  number  of  steps  taken  by  those  who 
successfully  completed  that  task)  for  Experiment  1 


VERSION  OF  MANUAL 

Elaborated  Unelaborated 
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(1.15) 

i 
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1 
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The  numbers  in  parentheses  are  the  means  for  each  cell 
adjusted  for  the  degree  of  prior  computer  usage. 


TABLE  5c 


Efficiency  on  completed  tasks:  the  proportion  of  steps  subjects  took  to 
complete  tasks  relative  to  the  minimum  number  of  steps  required  to 
successfully  complete  those  tasks,  for  Experiment  1 


VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before  I  2.4  I  2.3  I  2J 
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Efficiency  of  task  execution:  time  on  task 

We  also  measured  how  long  subjects  took  to  perform  a  task.  Time  was  measured 
using  the  computer-generated  timestamps  for  when  a  command  was  issued.  This 
measure  is  noisy  since  subjects  felt  under  no  time  pressure  and  were  not  told  that  we 
were  measuring  how  quickly  they  performed  the  tasks.  Table  6a  presents  the  mean 
time  subjects  took  to  do  all  of  the  required  sub-tasks,  as  a  function  of  condition.  These 
times  include  times  for  sub-tasks  that  subjects  did  not  successfully  complete.  There  is 
no  difference  in  total  time  as  a  function  of  whether  subjects  had  read  the  task 
instructions  prior  to  studying  the  manual  (the  BEFORE  vs.  AFTER  variable).  Although 
there  was  a  20%  difference  in  time  taken  to  complete  the  three  sub-tasks  as  a  function 
of  which  version  the  subjects  studied  (40  minutes  for  the  unelaborated  version  vs.  33 
minutes  for  the  elaborated  version),  this  difference  was  not  significant.  Again,  the  fastest 
condition  seemed  to  be  the  Elaborated-After  condition,  although  this  contrast  was  also 
non-significant. 

The  total  time  measure  suffers  from  the  same  biases  as  those  mentioned  above. 
Therefore,  we  computed  the  ratio  of  a  subject's  successful  completion  time  to  the  mean 
of  the  successful  completion  times  for  that  task.  We  took  the  arithmetic  mean  of  the 
ratios  for  each  subject.  These  values  are  displayed  in  Table  6b.  The  pattern  of  results 
is  in  the  same  direction  as  the  other  dependent  measures,  but  these  effects  are  also 


unreliable. 


Table  6a 


Before 
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subjects  to  perform  all  subtasks 
for  Experiment  1 
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taken  by  those  subjects  who  successfully  completed 

£ 

that  task)  for  Experiment  1 


VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before 

1 

1 

1 

.99 

(1.04) 

1 

1 

1 

1 

1.03 

(.96) 

After 

1 

1 

.96 

1 

1.16 

1 

1 

(.98) 

1 

1 

(1.15) 

.98 

1 

1.10 

(1.01) 

(1.06) 

The  numbers  in  parentheses  are  the  means  for  each  cell 
adjusted  for  the  degree  of  prior  computer  usage. 


Effects  of  Prior  Experience  on  Performance 


We  did  not  control  for  prior  experience  with  computers  in  this  first  experiment.6 
We  randomly  assigned  subjects  who  had  had  minimal  experience  with  computers  to 
conditions:  however,  we  did  note  computer  background  (by  having  all  subjects  complete 
a  self-rating  questionnaire),  and  used  this  information  to  perform  analyses  of  covariance. 

We  classified  experience  on  two  dimensions:  degree  of  prior  computer  usage  (rare 
frequent  and  know  multiple  operating  systems)  and  number  of  programming  languages 
studied  (none,  one  or  more  than  one  language).  We  performed  3x3  analyses  of 
variance  on  three  dependent  measures:  proportion  of  steps  per  completed  task  (as 
compared  to  the  mean  number  of  steps  taken  to  complete  each  task),  proportion  of  time 
taken  per  completed  task  and  percent  completion.  Subjects  who  had  studied  more 
programming  languages  completed  significantly  more  tasks  than  those  who  did  not  know 
any  languages,  F(2,30)  =  4.10,  pc. 05.  Subjects  who  rated  themselves  as  having  little 
experience  using  computers  took  (marginally)  significantly  more  time  than  other  subjects. 
F(2.30)  =  2.98,  pc.  10.  The  effects  of  degree  of  computer  experience  and  number  of 
programming  languages  did  not  have  additive  effects  for  the  number  of  steps  needed  to 
complete  a  task.  F(4,30)  =  3.05.  pc. 05.  No  other  effects  or  interactions  approached 
significance  for  these  dependent  variables 

We  performed  analyses  of  covariance  on  all  our  measures  using  each  covariate. 
These  analyses  showed  a  significant  negative  correlation  between  time  taken  to  perform 
the  tasks  and  rated  prior  use  of  computers.  F(  1.34)  =  4. 90.  pc. 05.  The  regression 
coefficient  was  -2.04.  There  was  also  a  negative  correlation  between  steps  taken  and 
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We  tried  using  subjects  who  had  very  little  experience:  however,  they  found  the  task  impossibly  tutfic  j  • 
and  uniformly  failed  the  task.  Therefore,  we  only  accepted  subjects  who  had  some  experience  vith 
computers,  though  none  with  micro-computers 
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this  variable,  F{1,34)  =  4.52,  p<  .05.  This  regression  coefficient  was  -5.13.  The  only  other 
reliable  correlation  was  between  the  number  of  programming  languages  known  and 
proportion  of  tasks  completed.  F(1,34)  =  7.22,  pc. 05.  (regression  coefficient  =  .  72).  The 
numbers  in  parentheses  in  Tables  3.  5b  and  6b  show  the  means  adjusted  for  the 
effects  of  prior  experience.  The  adjusted  mean  selected  was  always  from  the  covariate 
that  had  more  impact  on  that  analysis.  Although  there  were  significant  effects  due  to 
these  covariates,  the  adjustments  were  quite  slight  and  did  not  change  the  overall 
pattern  of  results  nor  the  significance  of  any  result.  Thus,  although  there  were  effects 
of  prior  experience,  they  were  not  confounded  with  assignment  of  subject  to  condition. 

Declarative  Test 

The  pencil  and  paper  test  was  administered  after  subjects  completed  the  on-line 
test.  This  test  was  also  administered  to  a  separate  group  of  16  C-MU  students  who 
took  this  paper  and  pencil  test  immediately  after  studying  the  manual,  and  did  not  do 
the  on-line  task.  (Half  of  these  16  subjects  read  the  elaborated  version  and  half  read 
the  unelaborated  version.) 

A  scoring  system  was  developed  for  the  two  kinds  of  questions  on  the  declarative 
test.  As  described  in  the  Methods  section,  half  of  the  declarative  knowledge  questions 
required  the  subject  to  write  a  command  to  produce  a  specified  result.  The  other  half 
of  the  questions  required  that  the  subject  describe  (in  writing)  what  effect  a  specified 
command  would  have  in  a  specified  context.  On  the  command-generating  questions, 
subjects  got  full  credit  if  they  gave  the  name  or  an  accepted  abbreviation  of  the 
appropriate  command,  and  gave  the  right  parameters  for  the  command  in  the  right 
order.  Subjects  were  credited  for  any  single  command  that  would  produce  the  desired 
effect,  i.e..  they  were  not  penalized  for  giving  more  path  information  than  was  absolutely 
necessary,  nor  for  giving  an  alternative  command  if  there  was  one.  To  receive  full  credit 
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on  the  description  questions,  subjects  had  to  "interpret"  each  part  of  the  supplied 
command.  For  example,  if  the  command  created  a  file  in  a  particular  location,  the 
subject  had  to  say  both  that  it  created  the  file  and  tell  where  it  would  appear.  (Partial 
credit  was  given  if  some  but  not  all  the  information  was  supplied.) 

Table  7  displays  the  results  for  the  paper  and  pencil  test  for  subjects  who  had 

first  completed  the  skill  test.  The  pattern  looks  very  much  like  the  procedural  tests: 
There  are  no  main  effects  or  interactions;  however,  the  Elaborated-After  condition  was 
significantly  better  than  the  other  three,  t(38)  =  2.26,  p<. 05.  One  reason  that  the  paper 

and  pencil  test  might  mirror  the  skill  test  is  that  subjects  had  already  practiced  the 

commands  in  the  first  test,  biasing  performance  in  favor  of  the  conditions  where  the 

correct  answers  had  been  practiced.  The  performance  of  those  subjects  who  did  not 
first  perform  on  the  skill  test  is  relevant  to  this  interpretation.  Those  subjects  who  read 
the  elaborated  version  got  64%  correct  on  the  test  and  those  who  read  the  shorter 
version  got  only  54%  correct.  That  difference  was  non-significant. 

The  fact  that  performance  was  5-10%  worse  for  subjects  who  did  not  first  perform 
the  skill  test  is  consistent  with  the  interpretation  of  a  bias  from  prior  testing.  Still,  the 
difference  between  the  elaborated  and  unelaborated  versions  was  even  bigger  (although 
not  significant)  for  the  subjects  who  only  took  the  declarative  test.  The  reason  that 
elaborations  may  have  helped  here  while  not  in  previous  studies  which  employed 
declarative  tests  is  that  the  paper  and  pencil  test  was  more  a  test  of  procedural 
knowledge  rather  than  declarative  knowledge,  required  subjects  to  generate  the 

appropriate  command  in  a  given  context  or  understand  and  generate  what  the  computer 
would  produce  in  a  given  context.  In  contrast,  the  declarative  tests  in  previous  studies 
tended  to  require  simple  fact  matching.  This  point  will  be  discussed  further  in  the  next 


section. 


TABLE  7 


Mean  percent  correct  for  paper  and  pencil  test 
for  Experiment  1 

VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before  I  .65  I  .63 


INSTRUCTION 


- 
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Discussion 

It  would  be  useful  at  this  point  to  summarize  our  findings.  Subjects  performed 
skill  tasks  better  if  they  had  studied  the  elaborated  version  of  the  manual,  but  this 
advantage  obtained  only  if  they  had  read  the  manual  without  knowing  what  the  test  of 
skill  would  be.  For  subjects  who  had  prior  knowledge  of  the  tasks,  the  converse  was 
true.  They  performed  well  only  if  they  read  the  unelaborated  version  of  the  manual.  We 
expected  the  elaborated  manuals  to  produce  better  performance,  and  that  hypothesis 
was  confirmed,  albeit  weakly.  We  also  expected  that  having  knowledge  of  the  task  prior 
to  study  would  facilitate  performance  overall,  but  that  did  not  occur. 


At  the  beginning  of  this  paper,  we  discussed  two  sources  of  elaborations  and 
contrasted  author-provided  elaborations  with  reader-generated  elaborations.  We  expected 
that  prior  knowledge  of  the  task  would  allow  the  reader  to  generate  better  and  more 
relevant  elaborations  and  to  focus  their  attention  on  those  aspects  of  the  manual  that 
would  be  most  relevant  to  the  subsequent  task.  Using  these  constructs,  it  is  possible  to 
explain  the  pattern  of  results  we  obtained. 


The  two  After  cells  most  closely  represent  the  conditions  of  the  numerous  studies 
on  author-provided  elaborations  cited  earlier.  We  found  that  subjects  consistently 
performed  best  in  the  Elaborated-After  condition  and  worst  in  the  Unelaborated-After 
condition.  Clearly,  in  the  skill-performance  domain,  subjects  perform  better  when 
instructional  texts  contain  author-provided  elaborations  than  when  they  do  not. 


It  is  important  to  bear  in  mind  that  subjects  performed  pretty  well  with  either 
version  of  the  manual.  The  subjects  in  the  worst  cell,  the  Unelaborated-After  condition, 
still  completed  three-quarters  of  the  tasks  successfully.  Furthermore,  the  Unelaborated- 
Before  condition  was  frequently  the  second-best  condition.  This  provides  some 
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independent  reassurance  that  the  difference  cannot  be  attributed  to  the  unelaborated 
version  missing  any  essential  information  for  completing  the  tasks. 

One  reason  why  advance  information  about  the  tasks  did  not  produce  better 
performance  overall  may  be  that  it  caused  subjects  to  focus  their  attention  overly 
selectively.  That  is.  they  inadvertently  ignored  too  much  relevant  information.  This  could 
occur  if  subjects  thought  they  understood  what  was  required  to  perform  the  specified 
tasks  (and  focussed  their  attention  accordingly),  but  were  wrong.  One  reason  the 
subjects  may  have  made  such  mistakes  is  that  the  task  instructions  were  taken  away 
prior  to  studying  the  manual,  i.e.,  subjects  were  not  allowed  to  refer  to  the  instructions 
while  reading.  Subjects  might  easily  have  misrer.  limbered  or  misinterpreted  the  nature 
of  the  tasks.  Another  related  problem  is  that  by  giving  the  task  instructions  before 
studying  the  manual  and  then  removing  them,  we  gave  subjects  in  the  Before  conditions 
an  additional  memory  load  (i.e.,  trying  to  remember  the  task  instructions)  that  subjects  in 
the  After  conditions  did  not  have.  This  too  could  cause  inferior  processing  of  the  new 
information. 

For  the  Elaborated-before  condition,  the  extra  source  of  elaborations  may  ultimately 
have  hampered  these  subjects  who  already  had  a  long,  elaborated  document  to  read. 
One  simple  explanation  for  the  poorer  performance  in  the  Elaborated-Before  condition  (as 
compared  to  the  Elaborated-After  condition)  is  that  it  took  subjects  too  long  to  read  the 
elaborated  version  of  the  manual  when  they  were  simultaneously  trying  to  generate  more 
task-specific  elaborations.  There  was  not  enough  time  to  read  the  entire  document 
thoroughly  under  these  constraints. 

Our  second  experiment  was  intended  to  eliminate  some  of  the  factors  that  might 


have  hampered  performance  in  the  Before  condition.  We  were  interested  to  see  whether 


the  elaborated  version  would  again  be  superior  to  the  unelaborated  version  and  whether 
the  Before  conditions  would  be  uniformly  better  than  the  After  conditions  when  some  of 
the  problems  mentioned  above  were  eliminated.  In  Experiment  2.  subjects  were  allowed 
to  set  their  own  pace  for  studying  the  manual  (within  a  one  hour  limit).  This  change  was 
intended  to  finesse  the  problem  of  subjects  having  too  little  time  to  generate  task- 
specific  elaborations  for  the  already  lengthy  elaborated  version.  We  also  eliminated  the 
additional  memory  load  and  the  potential  for  mis-remembering  the  nature  of  the  tasks  by 
allowing  subjects  in  the  Before  conditions  to  keep  the  task  instructions  while  studying  the 
manual. 

The  paper  and  pencil  test  was  also  redesigned  to  make  it  a  better  test  of 
declarative  knowledge  and  less  a  test  of  procedural  skills.  Modifications  were  also  made 
to  the  manuals  and  the  task  requirements  to  eliminate  problems  such  as  a  subject 
thinking  that  he  or  she  performed  a  task  successfully  when  in  fact  the  procedure  used 
was  wrong.  These  changes  will  be  discussed  in  more  detail  below. 


Experiment  2 

This  experiment  was  largely  a  replication  of  Experiment  1  with  several  major 
modifications:  study-time  was  self-paced,  task  instructions  could  be  read  during  the 

study  period  as  well  as  before  the  study  period,  and  the  paper  and  pencil  test  following 
the  skill  test  tapped  declarative  knowledge  more  than  procedural  knowledge. 

Method 

Subjects 

Forty-three  Carnegie-Mellon  University  undergraduates  participated  either  for  course 
credit  (to  satisify  a  requirement)  or  for  pay.  Subjects  received  $6  and  no  credits,  or  $3 
and  1  credit,  or  2  credits  for  participating.  Subjects  were  screened  before  being  allowed 
to  participate,  in  order  to  insure  that  all  subjects  were  fairly  familiar  with  CMU's  Dec-20 
computer  (which  is  used  regularly  by  the  entire  campus  community),  and  had  taken  no 
more  than  one  semester  of  a  computer  programming  course.  No  subject  had  ever 
worked  on  a  microcomputer. 

Design  and  Materials 

The  design  was  the  same  as  in  Experiment  1  except  that  we  did  not  administer 
the  paper  and  pencil  test  to  any  subjects  who  had  not  already  completed  the  on-line 
task.  Subjects  were  again  randomly  assigned  to  the  four  conditions  defined  by 
orthogonally  crossing  two  factors:  degree  of  elaboration  (elaborated  vs.  unelaborated)  and 
when  knowledge  of  task  was  acquired  (before  vs.  after  studying  the  manual). 


The  manuals  were  modified  slightly,  primarily  by  deleting  redundancy.  For 
example,  in  the  manuals  used  in  the  first  experiment,  an  explanation  of  how  to  specify  a 
path  through  the  directory  structure  was  repeated  whenever  a  new  command  was 
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introduced.  In  this  experiment,  the  definition  of  a  path  was  explained  once.  When  a 
path  was  used  as  part  of  the  parameters  that  define  a  command,  reference  was  made 
to  the  section  that  explained  paths  in  greater  detail.  The  shorter  version  was  also 
abridged  by  deleting  even  more  information  that  was  deemed  to  be  elaborative.  e.g.. 
reasons,  consequences,  restatements.  The  elaborated  version  in  Experiment  1  contained 
11,216  words;  after  modification,  the  corresponding  version  in  Experiment  2  contained 
10,605  words.  Similarly,  the  unelaborated  version  which  contained  5.019  words  in 
Experiment  1,  now  was  reduced  to  3,542  words.  So,  in  Experiment  1  the  ratio  of  the 
longer  to  the  shorter  version  was  approximately  2:1.  In  this  study,  the  ratio  became 
closer  to  3:1. 

There  were  also  a  few  modifications  in  the  tasks  required  of  subjects.  In  the  first 
experiment,  it  was  possible  for  subjects  to  successfully  complete  the  tasks  without  having 
to  negotiate  their  way  through  the  directory  structure.  One  modification  of  the  tasks  in 
this  study  was  to  force  subjects  to  draw  on  what  they  learned  about  subdirectories  by 
specifying  paths  to  subdirectories  in  their-commands,  or  changing  the  default  directory  to 
a  subdirectory.  Another  modification  was  to  reduce  repetitiveness  within  a  task.  For 
example,  subjects  did  not  have  to  rename  as  many  files  as  in  the  first  study  The  tasks 
are  described  in  more  detail  in  the  Procedure  section. 

The  paper  and  pencii  test  bore  little  resemblance  to  the  one  used  in  the  first 
experiment.  In  Experiment  1,  the  subject  was  asked  to  either  generate  a  command  that 
would  produce  the  described  result,  or  to  describe  what  the  result  would  be  of  issuing  a 
particular  command  in  a  particular  context.  Both  types  of  questions  required  subjects  to 
apply  their  general  knowledge  about  commands  to  specific  contexts  in  ways  very  similar 
to  what  was  required  for  the  nominally  more  "procedural"  on-line  task.  In  Experiment  2.. 
we  wished  to  make  the  pencil-and-paper  test  tap  declarative  knowledge  more  directly. 


The  new  test  consisted  of  24  true/false  questions  pertaining  to  the  concepts  in  the 
manual.  We  believe  that  these  questions  required  less  ability  to  recognize  contexts  of 
application  of  a  fact,  and  depended  more  on  the  ability  to  retrieve  studied  facts  from 
memory.  (Sample  questions  from  this  test  are  provided  in  the  Appendix),  in  order  to 
reduce  the  possibility  that  prior  knowledge  would  affect  performance  on  the  test,  we 
discarded  any  question  that  was  not  judged  at  chance  levels  by  subjects  who  had  not 
read  a  manual.  We  asked  43  people  to  answer  the  questions  and  discarded  any 
question  whose  accuracy  was  below  25  or  above  75  percent  correct. 

Procedure 

The  procedure  for  this  experiment  differed  slightly  from  that  used  in  Experiment  1. 
Those  subjects  assigned  to  one  of  the  Before  conditions  were  again  given  the  task 
instructions  to  read  prior  to  studying  the  manual;  however,  this  time  they  were  allowed  to 
keep  the  instructions  during  the  study  phase.  Subjects  assigned  to  the  After  conditions 
were  told  that  they  would  read  a  manual  about  using  the  PC  and  would  then  be  given 
a  series  of  tasks  to  do  on  the  computer. 

Both  groups  read  the  manual  in  front  of  the  PC.  Unlike  the  first  experiment  in 
which  all  subjects  studied  the  manuals  for  45  minutes,  subjects  were  now  given  up  to 
one  hour  in  which  to  read  the  entire  manual.  Subjects  were  instructed  to  notify  the 
expermienter  when  they  felt  they  had  mastered  the  manual  and  were  ready  to  proceed 
to  the  task.  The  experimenter  informed  the  subjects  when  half  the  hour  had  elapsed  and 
recorded  how  long  subjects  spent  studying  the  material.  Since  subjects  would  not  have 
the  manual  available  during  the  skill  test,  subjects  were  advised  to  review  the  important 
points  of  the  manual  until  they  felt  they  had  mastered  them  or  until  the  one  hour  time 


period  had  elapsed. 


When  subjects  finished  studying  the  material,  the  experimenter  returned  to  the 
room,  took  away  the  manual  and.  in  the  case  of  the  After  conditions,  gave  subjects  the 
task  instructions.  The  experimenter  remained  in  the  room  while  subjects  worked, 
watched  for  impasses  and  noted  any  interesting  strategies  that  would  be  missed  by  the 
on-line  recording  of  the  human-computer  interaction.  This  time  the  task  instructions  read 
as  follows: 

Instructions 

These  tasks  will  allow  you  to  practice  using  the  concepts  you  learned  from 
the  manual.  You  may  work  on  these  tasks  in  any  order.  Continue  working 
until  you  are  satisfied  that  you  have  completed  the  tasks  to  the  best  of  your 
abilities.  We  want  you,  however,  to  work  as  efficiently  as  possible. 

Task  1.  Before  you,  in  drive  A.  is  a  diskette  containing  a  number  of  files. 
Some  of  these  files  have  the  word  "PART"  in  their  names,  such  as  file 
"PART.l."  We  want  you  to  change  the  names  of  these  files.  The  new  name 
that  you  should  give  each  file  appears  as  the  first  line  of  that  file.  So.  inspect 
the  contents  of  each  file  that  now  has  "PART"  in  its  name  and  give  the  file 
the  name  that  you  find  on  the  first  line  of  the  file. 

Task  2.  Four  of  the  files  on  the  diskette  have  the  word  "DATA"  in  their 
names,  and  the  abbreviation  of  a  month  in  their  extension,  such  as  DATA. MAR. 

We  want  you  to  create  a  fifth  data  file  named  ALLDATA. 83  that  contains  the 
contents  of  the  other  four  data  files  appended  together.  Within  ALLDATA.83. 
the  files  should  appear  in  "chronological"  order:  that  is.  the  contents  of 
DATA. MAR  should  precede  the  contents  of  DATA.JUN  because  March  is  earlier 
in  the  year  than  June. 

Task  3.  Next,  you  should  create  two  subdirectories,  on  the  diskette  in  drive 
B.  One  subdirectory  is  to  be  named  PROGRAMS  and  the  other  named  DATA. 

Move  all  the  files  that  have  the  word  "Program"  in  their  names  from  drive  A 
into  the  PROGRAMS  directory  on  drive  B.  And.  similarly,  move  the  "Data"  files 
(including  ALLDATA.83  from  Task  2.  if  you  have  already  created  it),  into  the 
DATA  directory.  You  do  not  want  any  Program  or  Data  files  to  remain  on  the 
diskette  in  drive  A. 

Task  4  Finally,  you  should  eliminate  the  SOURCE  directory  and  everything  it 
contains  from  the  root  directory  of  the  diskette  in  drive  A.  The  root  directory  on 
drive  A  should  now  contain  only  a  list  of  files. 

Your  task  is  complete! 


After  the  four  tasks  were  completed  to  the  subject's  satisfaction,  the  surprise 


declarative  test  was  administered.  Subjects  were  allowed  to  take  as  long  as  they 
wanted  to  complete  this  paper  and  pencil  test;  however  they  generally  took  about  10 
minutes.  The  entire  experiment  typically  took  between  1.5  hours  and  2  hours. 

Results 

Data  were  analyzed  from  40  subjects,  although  43  subjects  actually  participated. 
Data  from  three  subjects  were  excluded  from  the  analyses,  two  due  to  computer  failure 
during  on-line  data  collection  and  one  due  to  a  subject  in  the  Unelaborated-Before 
condition  whose  protocol  revealed  that  he  had  worked  on  the  task  for  several  minutes 
during  the  study  period.  The  scoring  procedure  used  on  the  40  protocols  was  identical 
to  that  used  for  Experiment  1. 

Reading  Times 

Table  8  shows  the  mean  time  subjects  spent  studying  the  documentation,  as  a 
function  of  condition.  As  the  table  indicates,  subjects  took  more  time  to  read  the 
elaborated  version  than  the  unelaborated.  They  also  took  slightly  more  study  time  when 
they  had  prior  knowledge  of  the  tasks.  Although  these  differences  were  not  reliable, 
they  are  consistent  with  our  speculation  that  subjects  in  the  Elaborated-Before  condition 
of  Experiment  1  were  hurt  by  insufficient  time  to  both  process  the  author-provided 
elaborations  and  generate  their  own. 

in  order  to  assess  whether  the  one-hour  time  limit  was  itself  too  short,  we  noted 
how  many  subjects  used  all  or  most  of  the  alloted  time.  The  numbers  in  parentheses 
in  Table  8  represent  the  percentage  of  subjects  in  each  condition  who  studied  the 
material  for  more  than  50  minutes.  As  we  expected,  subjects  studying  the  elaborated 
version  of  the  manual  required  more  time  if  they  knew  the  task  instructions  in  advance. 
Even  though  the  mean  reading  times  for  the  two  elaborated  cells  look  nearly  identical. 
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TABLE  8 


Mean  Reading  Tine  in  minutes 
(Self-Paced) 
for  Experiment  2 


VERSION  OF  MANUAL 


Elaborated 

Unelaborated 

Before 

r 

i 

49 

1  45  1 

i 

L 

(.6) 

1  (.4)  1 

1  1 

After 

i 

48 

1  40  1 

I 

i 

(.3) 

1  (.1)  1 

1  1 

49 

1  43 

The  numbers  in  parentheses  are  the 

percentage  of  subjects  in  each  cell 
who  studied  the  manual  for  more  than 
50  minutes. 


Performance  on  Task 


Success  on  Tasks 

Completeness  scores  were  devised  for  the  four  subtasks  in  this  study  in  the  same 
mannner  as  in  Experiment  1. 

*  Rename  task.  Subjects  were  awarded  one  point  for  discovering  the  correct 
new  name  for  each  file  and  another  point  for  sucessfully  renaming  the  files. 

*  Combine  task.  Subjects  were  awarded  1  point  for  combining  the  files  in  the 
right  order. 

*  Make  Directory  task.  Subjects  were  awarded  two  points  for  sucessfully 
creating  two  subdirectories,  two  points  for  copying  the  appropriate  files  into 
each  one,  and  two  points  for  deleting  the  copied  files  from  the  root 
directory. 

*  Remove  Directory  task.  Subjects  were  awarded  two  points  for  deleting  the 
files  in  two  successively  deeper  subdirectories  and  two  more  points  for 
removing  each  of  these  subdirectories. 

Table  9  displays  the  mean  percentage  of  tasks  successfully  completed,  as  a 
function  of  condition.  In  contrast  to  Experiment  1,  the  Before  conditions  now  produce 
better  performance  than  the  After  conditions.  Like  Experiment  1,  the  elaborated  versions 
of  the  manual  produced  better  performance:  however,  all  of  these  effects  are  quite 
small,  i.e..  there  are  no  significant  main  effects  or  interactions. 

We  also  performed  a  2x2  analysis  of  covariance  on  the  completeness  scores,  with 
reading  time  as  the  covariate.  The  adjusted  cell  means  from  this  analysis  appear  in 
parentheses  in  Table  9.  The  adjustments  are  quite  small  for  the  most  part  and  do  not 
produce  any  reliable  main  effects.  However,  the  analysis  revealed  a  slight  negative 
correlation  between  reading  time  and  ability  to  complete  tasks  successfully.  F(1.35)  =  3.02. 
pc.l,  r  =  -0.01. 


TABLE  9 


Mean  proportion  of  tasks  successfully  completed 

e 

for  Experiment  2 


Before 

INSTRUCTION 

After 


VERSION  OF  MANUAL 


Elaborated 

Unelaborated 

1 

1  .82 

1 

1  .80 

_T 

1  .81 

1  (.83) 

1 

1  (.80) 

I 

1  ( .82) 

1 

f 

1  .78 

1  .78 

1  .78 

1  (.79) 

1 

1  (.75) 

1 

1  ( .77) 
J 

.80 

i  .79 

(.81) 

(.78) 

The  numbers  in  parentheses  are  the  means 

for  each  cell  adjusted  for  the  length  of 
time  the  subjects  studied  their  version 
of  the  manual. 


Efficiency  of  task  execution:  number  of  steps 

Table  10a  presents  the  data  for  the  mean  number  of  steps  needed  to  complete 
the  four  tasks,  as  a  function  of  condition.  The  differences  between  the  Before  and  After 
conditions  and  the  Elaborated  and  Unelaborated  conditions  were  not  significant.  The 
interaction,  on  the  other  hand,  is  significant  F(  1,36)  =  5. 88.  p<.05.  The  Elaborated-After 
condition  produces  the  best  performance,  while  the  Unelaborated-After  condition  produces 
the  worst.  As  in  Experiment  1,  the  Elaborated-After  condition  is  significantly  better  than 
the  other  three,  t(36)  =  -3.18,  pc. 01. 

Table  10b  displays  the  proportion  of  steps  taken  per  completed  task.  As  in  the 
previous  experiment,  this  measure  compares  subjects'  efficiency  on  tasks  they  completed 
relative  to  other  successful  subjects  (the  ratio  involves  the  mean  number  of  steps  needed 
to  successfully  complete  a  subtask).  This  corrected  measure  does  not  show  a  reliable 
advantage  for  the  elaborated  conditions:  however,  the  interaction  between  version  and 
instruction  is  again  significant,  F(1,36)  =  6.28,  pc. 05.  In  this  case,  however,  it  is  the 
Unelaborated-After  that  seems  different  (worse)  than  the  other  three  conditions. 
t(36)=  1.99.  'pc.  10. 

Finally,  we  compared  the  number  of  steps  each  subject  took  to  complete  a 
subtask  against  the  minimum  number  of  steps  required  to  complete  that  task.  Table 
10c  presents  the  mean  proportion  of  steps  per  completed  task.  As  in  the  two  previous 
analyses,  this  measure  produced  no  reliable  main  effects,  but  the  interaction  between 
version  and  instruction  was  once  again  significant.  F(  1.36)  =  5.49.  pc  .05. 


TABLE  10a 


Mean  number  of  steps  taken  by  subjects  to 
perform  all  subtasks  in  Experiment  2 


Before 

INSTRUCTION 


After 


VERSION  OP  MANUAL 

Elaborated  Unelaborated 


1 

1 

1 

1 

93 

1 

1 

1 

1 

81.7 

IB 

1 

1 

1 

66.7 

1 

i 

1 

100.5 

1 

1  83.60 

1 

1 

79.85 

1 

91.1 

TABLE  10b 


Mean  proportion  of  steps  taken  per  successfully 
completed  task  (compared  with  the  mean  number  of 
steps  taken  by  those  who  successfully  completed 
that  task)  for  Experiment  2 

VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before 


INSTRUCTION 

After 


I 

1 

1 

1 

1.02 

(1.04) 

- 1 - 

1 

1 

1 

.84 

(.84) 

1  -93 

1 ( .94) 
1 

.89 

1 

1.18 

1  1.0* 

1 

1 

(.90) 

1 

1 

(1.15) 

Id.o; 

l 

.96 

1 

1.01 

(.97)  M-00) 


*  The  numbers  in  parentheses  are  the  means 

for  each  cell  adjusted  for  the  length  of 
time  the  subjects  studied  their  version 
of  the  manual. 
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TABLE  10c 

Efficiency  on  completed  tasks:  the  proportion  of  steps  subjects  took  to 
complete  tasks  relative  to  the  minimum  number  of  steps  required  to 
successfully  complete  those  tasks,  for  Experiment  2 


Before 

INSTRUCTION 

After 


VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Efficiency  of  task  execution:  time  on  task 

The  time  needed  to  complete  the  tasks  was  the  third  measure  of  skill  performance. 
Table  1 1  a  presents  the  mean  time  subjects  spent  working  on  the  specified  tasks  (i.e.  the 
time  until  they  either  successfully  completed  all  the  tasks  or  gave  up  on  completing 
some  or  all  of  them),  as  a  function  of  condition.  Here  the  time  taken  to  complete  the 
four  tasks  is  significantly  shorter  (20%  faster)  for  subjects  who  studied  the  elaborated 
manual  than  for  those  who  studied  the  unelaborated  version.  F(1.36)  =  4.64.  pc  .05. 
Subjects  in  the  Before  conditions  are  10%  or  3  minutes  faster  than  those  in  the  After 
conditions;  however  this  effect  and  the  interaction  are  non-significant.  Once  again,  the 
Elaborated-After  condition  is  superior  to  the  other  three,  t(36)  =  -2.30,  pc. 05,  and  the 
Unelaborated-After  is  much  worse,  t(36)  =  2.17,  p<  .05.  * 


Table  11b  presents  a  slightly  different  picture  of  efficiency  in  terms  of  time  spent 
on  task.  This  measure  only  includes  times  for  subtasks  that  are  successfully  completed 
and  computes  the  proportion  of  time  a  subject  spent  completing  a  subtask  relative  to 
the  mean  of  successful  completion  times  for  those  tasks.  Here  there  is  a  main  effect  of 
instruction.  F(1 .36)  =  4.43.  pc. 05.  such  that  subjects  in  the  Before  conditions  complete 
tasks  significantly  faster  than  subjects  in  the  After  conditions.  The  advantage  of  the 
elaborated  conditions  is  now  non-significant  because  subjects  in  the  Uneiaborated-Before 
condition  do  so  well.  In  contrast,  the  Unelaborated-After  condition  is  the  worst  condition, 
producing  a  marginally  significant  interaction  F(  1 .36)  =  3.47.  pc.  10.  Indeed  the 
Unelaborated-After  cell  is  significantly  worse  than  the  other  three.  t(36)  =  3.3,  pc. 01. 


TABLE  11a 


Mean  tine  (in  minutes)  taken  by  subjects 
to  perforn  all  subtasks  in  Experiment  2 

VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before  I  31-02  I  32.49 


INSTRUCTION 


TABLE  11b 


Mean  proportion  of  time  taken  per  successfully 
completed  task  (compared  with  the  mean  time 
taken  by  those  who  successfully  completed  that 
task)  for  Experiment  2 * 


VERSION  OF  MANUAL 

Elaborated  Unelaborated 


INSTRUCTION 


(.98) 


(1.09) 


Before 

1 

1 

.97 

- 1 - 

1 

•  92 

1  .95 

1 

1 

(.97) 

1 

1 

(.92) 

I  (.95) 

1 

After 

1 

.99 

1 

1.25 

1  1.12 

1 

1 

(.99) 

1 

1 

(1.25) 

1(1.12) 

1 

.98 

1 

1.09 

’.V 

r'- 

« 

y. 

r; 

t’ 


The  numbers  in  parentheses  are  the  means 

for  each  cell  adjusted  for  the  length  of 
time  the  subjects  studied  their  version 
of  the  manual. 


Lviv 


Declarative  Test 


Table  12  presents  the  mean  percent  correct  on  the  true/false  test  that  subjects  in 
the  four  conditions  took  after  completing  the  skill  test.  There  was  a  main  effect  of 
when  subjects  were  given  the  task  instructions,  F(  1.36)  =  5. 19,  p<  .05.  such  that  subjects 
performed  best  on  the  declarative  test  if  they  had  not  seen  the  task  instructions  prior  to 
reading  the  material  (the  After  condition).  Apparently  subjects  in  the  Before  condition 
who  read  selectively  for  the  specified  task  could  not  perform  as  well  on  an  unexpected 
test  for  which  there  was  no  prior  knowledge  of  the  questions  that  would  be  asked. 
There  was  no  significant  effect  of  the  degree  of  elaborations,  however  the  Elaborated- 
After  was  better  than  the  other  three  conditions.  t(36)  =  2.28,  p<  05,  suggesting  that 


success  with  the  skill  task  may  have  translated  to  the  declarative  task. 


TABLE  12 


Mean  percent  correct  for  true/falae 
test  in  Experiaent  2 

VERSION  OF  MANUAL 

Elaborated  Unelaborated 


Before  I  .73  I  *69 


INSTRUCTION 


General  Discussion 

The  data  from  the  second  study  tend  to  support  the  findings  of  Experiment  1. 


First,  skill  performance  is  better  for  subjects  who  studied  the  elaborated  version  of  the 
manual,  especially  if  they  did  not  have  advance  knowledge  of  the  tasks  they  would  have 
to  perform.  An  over-all  advantage  for  the  elaborated  version  appears  in  the  marginals 
for  every  table  in  this  paper.  Often  this  effect  is  weak  because  the  8efore  conditions  do 
not  show  this  advantage  for  the  elaborated  version  consistently.  Second,  we  expected 
an  advantage  to  accrue  from  knowing  about  the  tasks  prior  to  studying  the  manual.  That 
effect  emerged  as  well,  although  it  too  was  weak.  Prior  knowledge  of  the  tasks  appears 
to  be  a  mixed  blessing:  it  greatly  aids  subjects  studying  the  unelaborated  version,  while 
it  tends  to  impede  the  performance  of  subjects  studying  a  document  that  is  already 
elaborated. 

A  different  way  to  capture  our  results  is  to  note  that  the  aberrant  cell  in  both 
studies  is  the  Elaborated-Before  condition.  We  found  that,  in  general,  prior  knowledge  of 
the  task  helps  performance.  We  also  found  that,  in  general,  author-provided  elaborations 
help  performance.  However,  when  both  are  provided  together  in  the  Elaborated-Before 
condition,  performance  is  worse  than  one  would  expect.  In  the  second  study,  we 
attempted  to  change  this  pattern  by  reducing  the  memory  burden  in  the  Before 
conditions,  viz.,  by  allowing  subjects  to  keep  the  task  instructions  while  studying  the 
manual.  Although  subjects  in  this  condition  now  completed  more  of  the  task 

successfully,  they  did  not  maintain  their  advantage  on  other-  measures.  If  memory  load 
had  been  a  problem  in  Experiment  1.  it  clearly  was  not  the  primary  reason  why  the 
Elaborated-Before  condition  was  worse.  We  also  tried  to  improve  performance  in  that 
condition  by  giving  subjects  more  study  time.  We  raised  the  maximum  amount  of  time 
allowed  from  45  minutes  to  one  hour.  This  may  not  have  been  enough  additional  time. 


Indeed,  six  of  the  10  subjects  in  the  Elaborated-Before  condition  studied  the  manual  for 
over  50  minutes,  and  of  these  four  took  all  the  time  that  was  allowed. 

From  another  perspective,  our  results  are  not  that  surprising.  Consider  what  effect 
we  thought  prior  knowledge  of  the  task  would  have.  We  imagined  that  prior  knowledge 
would  allow  subjects  to  selectively  attend  to  the  material  within  the  document  that  is 
directly  relevant  to  the  task.  Time  or  attention  that  would  ordinarily  be  devoted  to  less 
relevant  information  could  be  reallocated  to  information  that  is  essential  for  performing 
the  specified  set  of  tasks.  This  extra  time  might  then  be  used  both  for  elaborating  on 
the  task-relevant  statements  in  the  document  and  for  rehearsing  these  critical  facts. 
Although  these  reader-generated  elaborations  are  intended  to  strengthen  memory  traces 
and  create  redundancies  that  mitigate  against  forgetting,  they  could  well  be  counter¬ 
productive  when  elaborations  are  already  present  in  the  document. 

Elaborations  are  often  thought  to  be  a  good  memory  aid  because  they  provide 
redundant  memory  traces.  If  any  one  fact  is  forgotten,  there  will  be  others  that  can  be 
retrieved,  or  else,  the  relevant  fact  can  be  reconstructed  from  the  various  traces  that  are 
still  available.  Elaborations  provide  additional  retrieval  routes  to  the  information  by 
making  more  connections  with  prior  knowledge.  In  spite  of  these  potential  benefits, 
elaborations  might  also  have  a  detrimental  effect  on  retrieval:  there  is  considerable 
evidence  that  interference  can  occur  from  having  too  many  related  facts  in  memory. 
There  may  be  a  ceiling  in  maximal  benefit  due  to  elaborations  and  that  once  that  level 
is  exceeded,  the  detrimental  effects  of  interference  come  into  play.  Ballstaedt  and  Mandl 
(1981)  suggest  that  subjects  who  generate  too  few  elaborations  and  subjects  who 
generate  too  many  have  poorer  retention  than  those  who  elaborate  an  intermediate 
amount.  Perhaps  that  result  is  due  to  the  interaction  of  redundancy  and  interference. 


There  is  another  reason  why  the  Elaborated-Before  condition  might  have  produced 
relatively  poor  performance.  Even  with  the  additional  study  time,  subjects  may  not  have 
had  sufficient  time  to  read  the  entire  manual  and  still  have  time  to  rehearse  the  task¬ 
relevant  points.  While  reading  the  document  the  first  time,  these  subjects  may 
conjecture  as  to  the  relevance  of  a  particular  command  for  performing  one  of  the  tasks. 
However,  rather  than  committing  that  command  to  memory,  the  subject  might  decide  to 
scan  the  entire  document  to  check  that  there  is  not  a  superior  command  described 
later.  The  reader  s  intention  would  be  to  return  to  that  command  later  if  no  better  one 
presented  itself.  The  problem  with  such  a  strategy  is  that  there  probably  was  not 
enough  time  to  make  the  planned  second  pass  through  the  document,  and  the  first  pass 
was  not  thorough  enough  to  give  these  subjects  a  deep  understanding  of  everything  they 
read.  Subjects  in  the  Unelaborated-Before  condition  might  well  adopt  this  same  strategy: 
however,  since  there  was  so  much  less  to  read  in  their  version,  they  were  able  to  make 
that  second  pass  through  the  manual  and  reap  the  benefits  of  focussing  more  attention 
on  the  information  relevant  to  the  task.  Conversely,  subjects  in  the  After  conditions  are 
unlikely  to  adopt  such  a  strategy  because  without  advance  knowledge  of  they  task,  they 
have  no  way  of  deciding  which  commands  are  more  relevant  than  others.  Their  strategy 
on  the  first  pass  would  be  to  devote  equal  attention  to  all  the  commands,  and  try  to 
commit  all  of  them  to  memory.  For  the  subjects  in  the  Elaborated-After  condition,  this 
pays  off  because  they  can  take  advantage  of  the  author-provided  elaborations.  The 
subjects  in  the  Unelaborated-After  condition,  however,  are  less  fortunate.  They  do  not 
have  author-provided  elaborations  to  help  them  attain  a  better  understanding  of  the 
commands  in  general,  and  they  cannot  generate  good  elaborations  on  their  own  because 
they  don't  know  what  kind  of  tasks  to  expect. 

Prior  knowledge  of  the  task  only  facilitated  skill  performance  with  unembellished 


manuals.  Further,  the  results  of  the  declarative  test  strongly  suggest  that  prior  knowledge 
of  the  procedural  task  hurt  subsequent  performance  on  the  surprise  declarative  test. 
This  is  consistent  with  past  research  that  found  performance  better  for  subjects  who 
were  told  what  questions  to  expect,  but  worse  for  those  same  subjects  on  unexpected 
questions  (e.g.,  Frase  1975:  Frase  1967).  Conceivably,  performance  on  unexpected  skill 
tasks  would  also  suffer  in  the  Before  condition. 

In  conclusion,  we  believe  that  we  have  found  evidence  that  carefully  chosen 
elaborations  make  manuals  more  effective  for  teaching  people  how  to  perform  a 
technical  skill.  It  is  clear,  however,  that  if  a  person  knows  in  advance  what  tasks  will 
need  to  be  done,  a  short,  summary-like  manual  is  more  appropriate.  This  result  should 
not  be  surprising  given  our  reasons  for  believing  that  elaborations  would  help  skill 
performance  but  detract  from  declarative  learning.  We  argued  that  the  advantage  of 
elaborations  would  come  primarily  from  helping  a  user  to  figure  out  which  of  the  various 
procedures  to  apply  in  a  specific  task  situation.  However,  with  prior  knowledge  of  task 
demands,  the  difficulty  of  figuring  out  which  of  the  studied  procedures  are  appropriate  is 
diminished.  While  reading  the  manual,  the  learner  can  figure  out  which  procedures  to 
use.  and  when  and  where  to  apply  them.  The  elaborations  that  the  learner  generates  will 
be  closely  related  to  the  tasks,  and  far  better  than  those  that  can  be  provided  by  the 
author  of  the  manual,  who  cannot  anticipate  all  possible  tasks,  let  alone  generate 
elaborations  to  handle  them. 
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APPENDIX 
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*  The  Commands  Taught  in  the  User’s  Manual 

*  Sample  Declarative  Test  Questions  --  Experiment  1 

*  Sample  Declarative  Test  Questions  --  Experiment  2 

*  Unanaiyzed  Excerpt  from  an  On-line  Protocol  of  a  Subject  s  interaction  with 
the  IBM-PC 


The  Same  Excerpt  after  Analysis  and  Segmentation  into  Subtasks 


The  Commands  Taught  in 
COMMAND 
DIR 
MKDIR 
CHDIR 
RMDIR 
TYPE 
RENAME 
COPY 

ERASE 

'-Numlock 

‘-Break 

d: 


the  User’s  Manual 
FUNCTION 

List  the  files  in  a  directory 

Create  (or  "make")  a  subdirectory 

Change  default  directory  assignment 

Eliminate  (or  "remove")  a  subdirectory 

Display  the  contents  of  a  file 

Change  the  name  of  a  file 

Create  a  duplicate  copy  of  a  file 
Combine  or  append  files  together 
Transer  data  between  system  devices 

Eliminate  a  file 

Freeze  the  screen 

Abort  the  current  command 

Change  the  default  drive  assignment 


Sample  Declarative  Test  Questions 
Experiment  1 


Suppose  that  the  following  directory  listing  shows  the  contents  of  the  root  directory 
on  the  diskette  in  drive  A  on  the  computer.  Answer  each  question  as  if  you  were 
starting  with  this  set  of  files,  and  write  your  answer  in  the  space  provided. 


Volume  in  drive  A  is  unlabelled 
Directory  of  A:\ 

MAIL  TXT  10368  1-20-83  9:00a 

RANDOM  LSP  1613  3-14-83  1 1 : 54p 

PROGRAMS  <DIR>  2-05-83  10:34a 

PARSER  LSP  795  2-24-83  3:15p 

4  File(s)  141312  bytes  free 


1.  What  does  the  following  command  do? 

At  the  A>  prompt,  type: 

COPY  RANDOM. LSP  +  PARSER. LSP  PROGRAMS\PROG.LSP  <CR> 


2.  Write  a  command  (specifying  parameters  as  necessary),  which 
would  accomplish  the  following: 

Create  a  subdirectory  called  "PROGS. LSP"  in  the  "PROGRAMS" 
subdirectory. 


Sample  Declarative  Test  Questions 
Experiment  2 

The  RMDIR  command  can  be  used  to  delete  any 

directory,  including  the  root  directory.  (True/False) 

To  signal  the  computer  that  you  are  specifying 
a  path,  you  must  begin  the  path  by  typing  a 

backslash  character.  (True/False) 

it  is  impossible  to  have  two  copies  of  the  same 

file,  by  the  same  name,  in  the  same  directory.  (True/False) 


Excerpt  from  an  On-Line  Protocol  (Unanalyzed) 
of  Subject’s  Interaction  with  the  IBM-PC 
Experiment  1 


[  12:09:23] [12:09:27]  A>dir 


Volume  In  drive  A  ham  no  label 
Directory  of  A : \ 


TIMER 

<DIR> 

2- 

•18-84 

12:08p 

SYS 

<DIR> 

2- 

•18-84 

12: lOp 

UNIT 

10 

470 

10- 

•18-83 

3:28p 

UNIT 

D 

396 

10- 

•  18-83 

3:51p 

UNIT 

T 

683 

10- 

-18-83 

3:53P 

MINCE 

BAT 

24 

10-18-83 

3 : 36p 

6  File(s)  149504  by tea  free 
C 1 2 : 09 : 30 J [ 12:09:36]  A>akdir  unit 

[12:09:36][ 12:09:55]  A>copy  unit.*  unit 

UNIT. 10 

UNIT.D 

UNIT.T 

3  File(a)  copied 
[  12: 10:03][ 12: 10:05]  A>dir\unit 


Volume  in  drive  A  haa  no  label 
Directory  of  A:\unit 


• 

<DIR> 

2 

•  18-84 

12: 12p 

•  • 

<DIR> 

2 

-18-84 

12: 12p 

UNIT 

10 

470 

10 

-18-83 

3:28p 

UNIT 

D 

396 

10 

•  18-83 

3 :5  Ip 

UNIT 

T 

683 

10 

-18-83 

3 : 53P 

5  File(s)  145408  by tea  free 
[  12: 10: 08 ] C 1 2 : 10:11]  A>dir 


Volume  in  drive  A  haa  no  label 
Directory  of  A:\ 


TIMER 

<DIR> 

2-18-84 

12:08p 

SYS 

<DIR> 

2-18-84 

•  12: lOp 

UNIT 

<DIR> 

2-18-84 

1 2 : 1 2p 

UNIT 

10 

470 

10-18-83 

3 : 28p 

UNIT 

D 

396 

10-18-83 

3:51p 

UNIT 

T 

683 

10-18-83 

3:53P 

MINCE 

BAT 

24 

10-18-83 

3:36p 

5 


7  File(s)  145*108  bytes  free 
[ 12: 10: 14][ 12: 10:21 3  A>del  unit.» 

C 12: 10:21 ][ 12: 10:253  A>dir 


Volume  in  drive  A 
Directory  of  A:\ 

has  no  label 

TIMER 

<DIR> 

2-18-84 

12:08p 

SYS 

<DIR> 

2-18-84 

12: lOp 

UNIT 

<DIR> 

2-18-84 

12: 12p 

MINCE 

BAT 

4  File(s) 

24  10-18-83  3 : 36p 

148480  bytes  free 

[12:10:28][12:12:13]  A>con: 

Bad  command  or  file  name 
[ 12: 12: 14][ 12: 12:37]  A>migrate.lst  eon: 
Bad  command  or  file  name 
[12:12:38][12:13:10]  A>create  migrate. 1st 
Bad  command  or  file  name 
[12:13:11][12:13:15]  A>dir 


Volume  in  drive  A 
Directory  of  A:\ 

has 

no  label 

TIMER 

<DIR> 

2-18-84 

SYS 

<DIR> 

2-18-84 

UNIT 

<DIR> 

2-18-84 

MINCE 

BAT 

24 

10-18-83 

1 2 ; 08p 

12: lOp 

12: 12p 

- -  3:36p 

*»  Fiie(s)  148480  bytes  free 

[ 12: 13: 18][ 12: 13:26]  A>del  aince.bat 


[ 12: 13: 26 ][ 12: 13: 30 ]  A>dir 


Volume  in  drive  A  has  no  label 
Directory  of  A:\ 


TIMER  <DIR> 

SYS  <DIR> 

UNIT  <DIR> 

3  File(s) 


2-18-84  12:08p 

2-18-84  12: 10p 

2-18-84  12: 12p 
149504  bytes  free 


C 12: 13i 33 JC 12: 14: 15]  A>copy  con:  migrate. 1st 


mince.bat 


1  File(s)  copied 
C 12: 14:42]C 12: 14:44]  A>dir 


Volume  in  drive  A  has  no  label 
Directory  of  A:\ 


TIMER  <DIR> 
SYS  <DIR> 
UNIT  <DIR> 
MIGRATE  LST 

4  File(s) 


2-18-84  12:08p 

2-18-84  12: lOp 

2-18-84  12:1 2p 
11  2-18-84  12: I6p 

148480  bytes  free 


1 12: 14:47][ 12: 14:52]  A>type  migrate. 1st 


mince.bat 


Excerpt  from  an  On-Line  Protocol 
After  Analysis  and  Segmentation  into  Subtask 
Experiment  1 


<MKD  Task>[Dir  ]  69  -(  71)  [12:09:27]  A>DIR 

<MKD  Task>[Mkdir  ]  70  -(  72)  [12:09:36]  A>MKDIR  UNIT 

<MKD  Task>[Copy  ]  71  -(  73)  [12:09:55]  A>C0PY  UNIT.*  UNIT 

<MKD  Task>[Dir  ]  72  -(  74)  [12:10:05]  A>DIR\UNIT 

<MKD  Task>[Dir  ]  73  -(  75)  [12:10:11]  A>DIR 

<MKD  Task>[ Delete]  74  -(  76)  [12:10:21]  A>DEL  UNIT.* 

<MKD  Task>[Dir  ]  75  -(  77)  [12:10:25]  A>DIR 

<MGT  Task>[Misc  ]  76  -(  78)  [12:12:13]  A>C0N: 

GT  Task>[Misc  ]  77  -(  79)  [12:12:37]  A>MIGRATE.LST  CON: 

<MGT  Task>[Mlsc  ]  78  -(  80)  [12:13:10]  A>CREATE  MIGRATE. LST 

<MGT  Task>[Dir  ]  79  -(  81)  [12:13:15]  A>DIR 

<MGT  Task>[Delete]  80  -(  82)  [12:13:26]  A>DEL  MINCE . BAT 

<MGT  Task>[Dir  ]  81  -(  83)  [12:13:30]  A>DIR 

<MGT  Task>[Copy  ]  82  -(  84)  [12:14:15]  A>C0PY  CON:  MIGRATE. LST 

<MGT  Task>[Dlr  ]  83  -(  85)  [12:14:44]  A>DIR 

<MGT  Task>[Type  ]  84  -(  86)  [12:14:52]  A>TYPE  MIGRATE. LST 
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1  Dr.  Robert  Sasaor 
U.  S.  Arey  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  DR.  ROBERT  J.  SEIDEL 
US  Any  Research  Institute 
5001  Eisenhower  Ave. 

300  N.  WASHINGTON  ST. 

Alexandria,  VA  22333 
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Air  Fore* 

1  U.S.  Air  Foret  OFFice  oF  ScientiFic 
Research 

Lire  Sciences  Directorate,  WL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Dr.  Earl  A.  Alluisi 
HQ.  AFHRl  1AFSC) 

Brooks  AFB,  T»  78233 

1  Bryan  Cal lean 
AFHRL/LRT 

Lowry  AFB,  CO  B0230 

1  Dr.  Genevieve  Haddad 
Prograe  Manager 
Lift*  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

1  Dr.  T.  H.  Longridge 
AFHRL /OTE 

Hilliaes  AFB,  AI  85224 

1  Dr.  John  Tangney 
AFOSR/NL 

Bolling  AFB,  DC  20332 

1  Dr.  Joseph  Yasatuke 
AFHRL/ LRT 

Lowry  AFB,  CO  80230 


Departrent  oF  DeFense 

1  Dr.  Craig  I.  Fields 
Advanced  Research  Projects  Agency 
1400  Wilson  Blvd. 

Arlington,  VA  22209 

1  Military  Assistant  For  Training  and 
Personnel  Technology 
OFFice  oF  the  Under  Secretary  oF  DeFens 
For  Research  it  Engineering 
Root  3C129,  The  Pentagon 
Washington,  DC  20301 

1  Major  Jack  Thorpe 
DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 

1  Dr.  Robert  A.  Wisher 
OUSDRE  (ELS) 

The  Pentagon,  Root  3D129 
Washington,  DC  20301 
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Civilian  Agencies 
1  Eduard  Esty 

Department  of  Education,  OERI 

ns  40 

1200  19th  St.,  NW 
Washington,  DC  20208 

1  Dr.  Arthur  Melaed 
724  Broun 

U.  S.  Dept,  of  Education 
Washington,  DC  20208 

1  Dr.  Andre*  R.  Molnar 
Office  of  Scientific  and  Engineering 
Personnel  and  Education 
National  Science  Foundation 
Washington,  DC  20530 

1  Dr.  Raasay  W.  Selden 
National  Institute  of  Education 
1200  19th  St.,  NW 
Washington,  DC  20208 

1  Dr.  Ed*ard  C.  Weiss 
National  Science  Foundation 
1800  6  Street,  NN 
Washington,  DC  20550 

1  Dr.  Frank  Withro* 

U.  S.  Office  of  Education 
400  Maryland  Ave.  3N 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Meaory  t  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  2C550 
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Private  Sector 

1  Dr.  John  R.  Anderson 
Departaent  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  13213 

1  Patricia  Baggett 
Departaent  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Eva  l.  Baker 
Director 

UCLA  Center  for  the  Study  of  Evaluation 
143  Moore  Hall 

University  of  California,  Los  Angeles 
Los  Angeles,  CA  90024 

l  Mr.  Avron  Barr 
Departaent  of  Coaputer  Science 
Stanford  University 
Stanford,  CA  94335 

1  Dr.  John  Black 
Yale  University 
Box  1 1 A ,  Yale  Station 
Ne*  Haven,  CT  06520 

1  Dr.  John  S.  Brown 
XEROX  Palo  Alto  Research  Center 
3333  Ccyote  Rnd 
Palo  Alto,  CA  94304 

1  Dr,  Bruce  Buchanan 
Departaent  of  Coaputer  Science 
Stanford  University 
Stanford,  CA  94335 

1  Or.  Jaiae  Carbonell 
Carnegie-Mellon  University 
Departrent  of  Psychology 
Pittsburgh,  PA  15213 

1  Dr.  Pat  Carpenter 
Depirtaent  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Davida  Chaney 
Departaent  of  Psychology 
Carnegie-Melon  University 
Schneley  Park 
Pittsburgh,  PA  15213 
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Private  Sector 

l  Euqene  Charniak 
Department  of  Computer  Science 
Brown  University 
Providence,  RI  02912 

1  Dr.  Hicheline  Chi 
Learning  R  l  D  Center 
University  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  13213 

1  Dr.  Mi 11 i a*  Clancey 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94306 

1  Dr.  Hictiael  Cole 
University  of  California 
at  San  Diego 

Laboratory  of  Comparative 
Human  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Allan  H.  Collins 
Bolt  Beranek  6  Newman,  Inc. 

50  (loulton  Street 
Cambridge,  HA  02138 

1  Dr.  Emmanuel  Donchin 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Dr.  Thomas  H.  Duffy 
Department  of  English 
Carnegie-Nelion  University 
Schenley  Park 
Pittsburgh,  CA  15213 

1  Dr.  Anders  Ericsson 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CD  30309 

1  Dr.  Paul  Feltovich 
Department  of  Hedical  Education 
Southern  Illinois  University 
School  of  Hedicine 
P.Q.  Bo*  3926 
Springfield,  IL  62708 


Private  Sector 

1  Hr.  Nallace  Feurzeig 
Departrent  of  Educational  Technology 
Bolt  Beranek  It  Newman 
10  Boulton  St. 

Cambridge,  HA  02238 

1  Dr.  Dexter  Fletcher 
University  of  Oregon 
Departrent  of  Computer  Science 
Eugene,  OR  97403 

1  Dr.  John  R.  Frederiksen 
Balt  Beranek  6  Newman 
50  Houlton  Street 
Cambridge,  HA  02138 

1  Dr.  Hichael  Senesereth 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Dedre  6entner 
Bolt  Beranek  l>  Newman 
10  Moulton  St. 

Cambridge,  HA  02138 

1  Dr.  Robert  61aser 

Learning  Research  6  Development  Center 
University  of  Pittsburgh 
3939  O’Hara  Street 
PITTSBURGH,  PA  15260 

l  Dr,  Harvin  D.  Slock 
217  Stone  Hall 
Cornell  University 
Ithaca,  NY  14833 

1  Dr.  Josph  Goguen 
SRI  International 
333  Ravenswood  Avenue 
Henlo  Park,  CA  9&023 

1  DR.  JAKES  S.  SREENQ 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O’HARA  STREET 
PITTSBURGH,  PA  I 52! 3 

1  Or.  Henry  H.  Half f 
Halff  Resources 
4918  3’rd  Road,  North 
Arlington,  VA  22207 


rim,  MKi 


REDER/CHARNEY 


ll-Jul-84 


Page  7 


Private  Sector 

1  Dr.  Reid  Hastie 
Department  of  Psychology 
Northwestern  University 
Evanston,  !L  60201 

1  Or.  Barbara  Hayes-Rcth 
Departeent  of  Cosputer  Science 
Stanford  University 
Stanford,  CA  95303 

1  Or.  Joan  1.  Heller 
Graduate  Group  in  Science  and 
Hatheeatics  Education 
c/o  School  of  Education 
University  of  California 
Berkeley,  CA  9^720 

1  Dr.  Jaees  R.  Haffean 
Depart sent  of  Psychology 
University  of  Delaware 
Newark,  DE  19711 

1  Nelissa  Holland 
American  Institutes  for  Research 
1035  Thomas  Jef‘erson  St.,  N.d. 
Hashington,  OC  20007 

1  Dr.  Kristina  Hooper 
Corporate  Research,  ATARI 
1196  Borregas 
Sunnyvale,  CA  94086 

1  Dr.  Harcei  Just 
Department  of  Psychology 
Carneqie-Hellcn  University 
Pittsburgh,  PA  15213 

1  Dr.  David  Kieras 
Department  of  Psychology 
University  of  Arizona 
Tuscon,  AZ  33721 


Private  Sector 

1  Dr.  Jill  Larkin 
Departrent  of  Psychology 
Carnegie  Hellon  University 
Pittsburgh,  PA  15213 

1  Or.  Alan  Lesgold 
Learning  RiD  Canter 
University  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  15260 

1  Dr.  Jim  Levin 
University  of  California 
at  San  Diego 

Laboratory  f of  Comparative 
Human  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Harcia  C.  Linn 
Lawrence  Hall  of  Science 
University  of  California 
Berkeley,  CA  94720 

1  Dr.  Don  Lyon 
P.  0.  Bov  44 
Higley  ,  AZ  83236 

1  Dr.  Jay  HcClelland 
Departrent  of  Psychology 
HIT 

Cambridge,  HA  0213? 

1  Dr.  James  R.  Hiller 
ComputersThought  Corporation 
1721  Best  Plano  Highwav 
9lano,  TY  75075 

1  Dr.  Allen  Hunro 

Behavioral  Technology  Laboratories 
1843  E’eia  Ave.,  Fourth  Floor 
Redondo  Beach,  CA  90277 


1  Dr.  Halter  Ki ntsch 
Department  of  Psychology 
'Jni/ersity  of  Colorado 
Boulder,  CO  80302 


1  Dr.  Donald  A  Norman 
Cognitive  Science,  C-015 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 


1  Dr.  Pat  Langley 
The  Robotics  Institute 
Carnegie-Heilon  University 
Pittsburgh,  ?A  15213 


1  Dr.  Jesse  3rl ansky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 
Alexandria,  VA  22311 
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Private  Sector 

Private  Sector 

1  Or.  Nancy  Pennington 

1  Dr.  Roger  Schank 

it  ■ m 

University  of  Chicago 

Yale  University 

Graduate  School  of  Business 

Departaent  of  Coaputer  Science 

1101  E.  38th  St. 

P.0.  Bex  2158 

» 

Chicago,  IL  60637 

New  Haven,  CT  06520 

& 

1  Dr.  Ann  Piestrup 

1  Dr.  Balter  Schneider 

U| 

THE  LEARN I N6  COMPANY 

Psychology  Departaent 

545  Niddlefield  Road,  Suite  170 

603  E.  Daniel 

Nenlo  Park,  CA  94025 


1  Dr.  Steven  E.  Poltrock 
Bell  laboratories  2D-444 
600  Mountain  Ave. 

Hurray  Hill,  NJ  07974 


1  Dr.  Lynne  Reder 
Departient  of  Psychology 
Carnegi e-Bel  Ion  University 
Schenley  Park 
Pittsburgh,  PA  15213 


1  Dr.  Fred  Reif 
Physics  Departient 
University  of  California 
Berkeley,  CA  9^720 


1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O’Hara  Street 
Pictsburgh,  PA  1521 


l  Dr.  Jeff  Richardson 
Denver  Research  Institute 
University  of  Denver 
Denver,  CO  90208 


1  llary  S.  Riley 
Prograe  in  Cognitive  Science 
Center  for  Huaan  Inforaation  Processing 
University  of  California,  San  Diego 
La  Jolla,  CA  92093 


1  Sr.  Andrew  H.  Rose 
Aaerican  Institutes  for  Research 
1055  Thoaas  Jefferson  St.  NN 
Washington,  DC  20007 


1  Dr.  Milliaa  9.  Rouse 
Seorgia  Institute  of  Technology 
School  of  Industrial  6  Systeas 
Engineering 
Atlanta,  6A  30332 


Chaapaign,  IL  61820 


1  Dr.  Alan  Schoenfeld 
Hatheaatics  and  Education 
The  University  of  Rochester 
Rochester,  MY  14627 


1  Dr.  Ted  Shortliffe 
Coaputer  Science  Departient 
Stanford  University 
Stanford,  CA  94305 


1  Dr.  Edward  E.  Saith 
Bolt  Beranek  l  Newaan,  Inc. 
50  Boulton  Street 
Caabridge,  BA  92138 


1  Dr.  Eliott  Soloway 
Yale  University 
Departient  of  Coaputer  Science 
P.0.  Box  2158 
New  Haven,  CT  06520 


1  Dr.  Kathryn  T.  Spoehr 
Psychology  Departaent 
Brown  University 
Providence,  RI  92912 


1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  HA,  Yale  Station 
New  Haven.  CT  06520 


1  Dr.  Albert  Stevens 


10  Boulton  St. 
Caabridge.  BA  02238 


1  Dr.  David  Stone 
KAJ  Software,  Inc. 
3420  East  Shea  Blvd. 
Suite  161 
Pheonix,  AZ  8502B 
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Private  Sector 

1  DR.  PATRICK  SUPPES 
INSTITUTE  FOR  HATHEHAT 1 CAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  9^-303 

1  Dr.  Kilcuai  Tatsuoka 
Coaputer  Based  Education  Research  Lab 
252  Engineering  Research  Laboratory 
Urbana,  IL  61801 

1  Dr.  Perry  N.  Thorndyke 
Perceptronics,  Inc. 

5*5  NiddlePield  Road,  Suite  1*0 
Nenlo  Park,  CA  9*025 

1  Dr.  Douglas  Tonne 
Uni v.  o*  So.  California 
Behavioral  Technology  Labs 
18*5  S.  Elena  Ave. 

Redondo  Beach,  CA  90277 

1  Dr.  Kurt  Van  Lehn 
ter ox  PARC 

Coyote  Hill  Road 
Palo  Alto,  CA  9*30* 

1  Beth  barren 

Bolt  Beranek  k  Neman,  Inc. 

50  Noulton  Street 
Cambridge,  f!A  02138 

1  Dr.  Keith  T.  descour: 

Perceptronics,  Inc. 

5*5  Hiddlefield  Road,  Suite  1*0 
Nenlo  Park,  CA  9*025 

1  Dr.  Nike  billiaas 
IntelliBenetics 
124  University  Avenue 
Palo  Alto,  CA  9*301 


